From xiang.zhao at gamegou.com Tue Nov 3 00:10:48 2015 From: xiang.zhao at gamegou.com (=?utf-8?B?WmhhbyBYaWFuZw==?=) Date: Thu Nov 5 00:42:10 2015 Subject: [odb-users] PRAGMA_DB in preprocessor define Message-ID: Dear ODB Support, I'm currently using odb 2.4.0 windows version. I'm trying to put a "#pragma db column" in a preprocessor macro define so I can reuse it: #define ODB_COUNT_EXISTS(obj, key) \ PRAGMA_DB(view object(obj)) \ struct obj##_count \ { \ PRAGMA_DB(column("count(" + obj::key + ")")) \ std::size_t count; \ }; \ PRAGMA_DB(view object(obj)) \ struct obj##_exists \ { \ PRAGMA_DB(column("exists(" + obj::key + ")")) \ bool exists; \ }; ODB_COUNT_EXISTS(group_fans_record, key.user_id) However the odb compiler gives me two errors: 1> OdbSchema.hxx:1187:4: error: db pragma 'column' is not associated with a declaration 1> OdbSchema.hxx:1187:4: error: db pragma 'column' is not associated with a declaration I tried to expand the code with "gcc -E", and put the result back to the original code: # 1201 "OdbSchema.hxx" #pragma db view object(group_fans_record) # 1201 "OdbSchema.hxx" struct group_fans_record_count { # 1201 "OdbSchema.hxx" #pragma db column("count(" + group_fans_record::key.user_id + ")") # 1201 "OdbSchema.hxx" std::size_t count; }; # 1201 "OdbSchema.hxx" #pragma db view object(group_fans_record) # 1201 "OdbSchema.hxx" struct group_fans_record_exists { # 1201 "OdbSchema.hxx" #pragma db column("exists(" + group_fans_record::key.user_id + ")") # 1201 "OdbSchema.hxx" bool exists; }; Well this time odb generates what I need. Is there something I can do about it? Thanks Zhao Xiang From boris at codesynthesis.com Thu Nov 5 04:21:48 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 5 04:21:50 2015 Subject: [odb-users] PRAGMA_DB in preprocessor define In-Reply-To: References: Message-ID: Hi Zhao, Zhao Xiang writes: > I'm trying to put a "#pragma db column" in a preprocessor macro [...] > > [...] > > However the odb compiler gives me two errors: Yes this is a limitation/bug in the ODB compiler. The "position pragmas" (i.e., those that apply to the declaration that follow them) are implemented using source locations. For macros we happen to use the macro expansion point as the location of the pragmas inside the macro. And fixing this properly won't be simple. On the other hand, there is a pretty simple workaround. It makes the macro slightly more verbose but is probably not a big deal in this context: #include /* #define ODB_COUNT_EXISTS(obj, key) \ PRAGMA_DB(view object(obj)) \ struct obj##_count \ { \ PRAGMA_DB(column("count(" + obj::key + ")")) \ int count; \ }; */ #define ODB_COUNT_EXISTS(obj, key) \ struct obj##_count \ { \ int count; \ }; \ PRAGMA_DB(view(obj##_count) object(obj)) \ PRAGMA_DB(member(obj##_count::count) \ column("count(" + obj::key + ")")) \ #pragma db object struct test { #pragma db id int id; }; ODB_COUNT_EXISTS(test, id) Boris From xiang.zhao at gamegou.com Thu Nov 5 04:42:04 2015 From: xiang.zhao at gamegou.com (=?utf-8?B?WmhhbyBYaWFuZw==?=) Date: Thu Nov 5 04:43:28 2015 Subject: [odb-users] PRAGMA_DB in preprocessor define In-Reply-To: References: Message-ID: Hi Boris, Thanks for your help, your marco works like a charm. I tried do find a "#pragma db column" that does not depend on position, but failed due to unfamiliar with the manual. Regards Zhao Xiang ------------------ Original ------------------ From: "Boris Kolpackov"; Date: Thu, Nov 5, 2015 05:21 PM To: "Zhao Xiang"; Cc: "odb-users"; Subject: Re: [odb-users] PRAGMA_DB in preprocessor define Hi Zhao, Zhao Xiang writes: > I'm trying to put a "#pragma db column" in a preprocessor macro [...] > > [...] > > However the odb compiler gives me two errors: Yes this is a limitation/bug in the ODB compiler. The "position pragmas" (i.e., those that apply to the declaration that follow them) are implemented using source locations. For macros we happen to use the macro expansion point as the location of the pragmas inside the macro. And fixing this properly won't be simple. On the other hand, there is a pretty simple workaround. It makes the macro slightly more verbose but is probably not a big deal in this context: #include /* #define ODB_COUNT_EXISTS(obj, key) \ PRAGMA_DB(view object(obj)) \ struct obj##_count \ { \ PRAGMA_DB(column("count(" + obj::key + ")")) \ int count; \ }; */ #define ODB_COUNT_EXISTS(obj, key) \ struct obj##_count \ { \ int count; \ }; \ PRAGMA_DB(view(obj##_count) object(obj)) \ PRAGMA_DB(member(obj##_count::count) \ column("count(" + obj::key + ")")) \ #pragma db object struct test { #pragma db id int id; }; ODB_COUNT_EXISTS(test, id) Boris From markus at markusklemm.net Fri Nov 6 07:34:13 2015 From: markus at markusklemm.net (Markus Klemm) Date: Fri Nov 6 07:34:29 2015 Subject: [odb-users] [Feature Request] odb::query operator== and operator!= Message-ID: <62DEBCFF-3EEA-440A-B0F9-8F71C7453392@markusklemm.net> Hi there, please correct me if I'm wrong, but there seem to be no equality e.g. inequality operator defined for odb::query. This would be very helpful, to for instance, decide if loading of data is necessary. Best regards Markus Klemm From boris at codesynthesis.com Fri Nov 6 09:58:01 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Nov 6 09:58:03 2015 Subject: [odb-users] [Feature Request] odb::query operator== and operator!= In-Reply-To: <62DEBCFF-3EEA-440A-B0F9-8F71C7453392@markusklemm.net> References: <62DEBCFF-3EEA-440A-B0F9-8F71C7453392@markusklemm.net> Message-ID: Hi Markus, Markus Klemm writes: > please correct me if I'm wrong, but there seem to be no equality e.g. > inequality operator defined for odb::query. You are correct, these operators are not provided for odb::query. > This would be very helpful, to for instance, decide if loading of > data is necessary. While I can see how this can be useful, implementing these operators properly will be non-trivial, to say the least. For example, would you expect these two queries to compare equal? (query::name == "John" && query::age == 18) (query::age == 18 && query::name == "John") What about these two: !(query::name == "John" || query::name == "Jane") (query::name != "John" && query::name != "Jane") At the same time, it should be pretty easy to keep the data for your queries in some "canonical" form/order so that it is easy to decide whether the two queries are equivalent. Boris From paul.harrison at manchester.ac.uk Tue Nov 10 10:17:51 2015 From: paul.harrison at manchester.ac.uk (Paul Harrison) Date: Tue Nov 10 10:18:08 2015 Subject: [odb-users] problem with static linking Message-ID: Hi, I am trying to create a (mostly) statically linked executable that uses odb and mysql - my final link and error is g++ -Wl,-Bsymbolic -DLinux_2 -DNOTJIVE -DEMERLIN_DEBUG -g -fPIC -D_FILE_OFFSET_BITS=64 -I/home/emerlin/work/emerlin/include -I/usr/include -I/usr/include/mysql -I/usr/local/include -I/home/emerlin/work/emerlin/qwt/include -I/usr/include/QtGui -I/usr/include/QtCore -I/usr/include/QtNetwork -I/usr/include -I/home/emerlin/work/emerlin/CCfits/include -I/usr/include/cfitsio -I/home/emerlin/work/emerlin/include -DBROKER_HOST=\"emproc1\" -DEMERLIN data_tool.o -o data_tool -L/home/emerlin/work/emerlin/lib/Linux_2_x86_64 -Wl,-rpath,/home/emerlin/work/emerlin/lib/Linux_2_x86_64 -L/home/emerlin/work/emerlin/lib/Linux_2_x86_64 -L/usr/local/lib64 -L/usr/lib64/mysql -L/home/emerlin/work/emerlin/qwt/lib -L/usr/lib -L/home/emerlin/work/emerlin/CCfits/lib -L/usr/lib -L/usr/X11R6/lib -Wl,-Bstatic -ldh_common -lojd -larchive_db -ldata_manager -leop_db -lscat -ldata_exporters -ldelays -lslalib -lemerlin_msgs -lemerlin_codes -lmessage_base -lmessage_common -lconfigs_db2 -lconfigs_db -lpersist2 -lslalib -lCCfits -lcfitsio -lherbert -lpersist -lherbert -lstore_classes -lodb -lodb-mysql -lodb-boost -lodb -lsocket++ -lerrors -lutil -lmachine -llocal -lz -Wl,-Bdynamic -lmysqlclient -lpthread -lboost_serialization-mt -lboost_system-mt -lboost_filesystem-mt -lboost_date_time /usr/local/lib/libodb-mysql.a(connection-factory.o): In function `~mysql_thread_init': /opt/packages/odb/libodb-mysql-2.3.0/odb/mysql/connection-factory.cxx:79: undefined reference to `THR_KEY_mysys' /opt/packages/odb/libodb-mysql-2.3.0/odb/mysql/connection-factory.cxx:80: undefined reference to `THR_KEY_mysys' /opt/packages/odb/libodb-mysql-2.3.0/odb/mysql/connection-factory.cxx:79: undefined reference to `THR_KEY_mysys' /opt/packages/odb/libodb-mysql-2.3.0/odb/mysql/connection-factory.cxx:80: undefined reference to `THR_KEY_mysys' /usr/local/lib/libodb-mysql.a(connection-factory.o): In function `mysql_thread_init': /opt/packages/odb/libodb-mysql-2.3.0/odb/mysql/connection-factory.cxx:69: undefined reference to `THR_KEY_mysys' /usr/local/lib/libodb-mysql.a(connection-factory.o):/opt/packages/odb/libodb-mysql-2.3.0/odb/mysql/connection-factory.cxx:79: more undefined references to `THR_KEY_mysys' follow collect2: ld returned 1 exit status make[1]: *** [data_tool] Error 1 make[1]: Leaving directory `/home/emerlin/work/emerlin/data_tool/source/build/Linux_2_x86_64/data_tool? I have seen https://bugzilla.redhat.com/show_bug.cgi?id=846602, but my problem is that I am compiling on scientific linux 6.7, which has a rather old version of mysql as standard - before I try to get a more up-to-date version installed, I thought that I would just ask if there is another solution to this problem? Regards, Paul. From boris at codesynthesis.com Tue Nov 10 10:37:29 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Nov 10 10:37:27 2015 Subject: [odb-users] problem with static linking In-Reply-To: References: Message-ID: Hi Paul, Paul Harrison writes: > /usr/local/lib/libodb-mysql.a(connection-factory.o): In function > `~mysql_thread_init': > /opt/packages/odb/libodb-mysql-2.3.0/odb/mysql/connection-factory.cxx:79: > undefined reference to `THR_KEY_mysys' It is strange that you get this error since ODB tries to detect whether this symbol is visible. Perhaps you built libodb-mysql and linking your application using different libmysqlclient libraries? Check the config.h file in libodb-mysql/odb/mysql/details/ for LIBODB_MYSQL_THR_KEY_VISIBLE. If it is defined, then the link test (see config.log for the command line) somehow manages to "see" this symbol while your command line does not. Boris From cfdx at list.ru Thu Nov 12 08:55:11 2015 From: cfdx at list.ru (=?UTF-8?B?0JTQsNC90LjQuNC7INCh0LDQstGH0LXQvdC60L4=?=) Date: Thu Nov 12 08:55:21 2015 Subject: [odb-users] Combining views queries conditions Message-ID: <1447336511.152752731@f361.i.mail.ru> Good day. Could you please tell me if it is a correct way to use ODB: ? #pragma db object class Object?; ? #pragma db view object(Object) class View?; ? void f( const query& q )? { ? ?auto res = db->query( q ); } ? I have tried it and it looks like it works. But is it a hack? In more complicated cases: #pragma db view object(Object1) object(Object2:...) class View2?; ? void f( const query& q )? { ? ?auto res = db->query( q ); } OR void f( const query& q1, const query& q2 )? { ? ?auto res = db->query( q1 && q2 ); } OR void f( const query& q1, const query& q2 )? { ? query qx = q1 && q2; // strange, but works! ? ?auto res = db->query( qx ); } And the most crazy use case: void f( const query& q )? { ? ?// works if View1 query condition uses a subset of objects, used in View2 ? ?auto res = db->query( q ); } ? Is combining of condition for different Object and Views queries a correct way of creation of condition for Views. And if ?yes?, what are the formal limits of applicability. I understand that it is possible to set conditions right in terms of the View fields, but I want to encapsulate creation subconditions in different parts of application, and reuse these parts in different places where different Views are used. Or maybe there exist recommended way (some best practices) to do this. Daniil Savchenko ? ?????????, ?????? ?. ???????? From gratta at visuol.com Thu Nov 12 08:12:46 2015 From: gratta at visuol.com (Florent GRATTA) Date: Thu Nov 12 09:52:14 2015 Subject: [odb-users] Exception : "object not persistent" when trying to load object which does not exist in database Message-ID: <4b577fabf112659aeadc19492484b594@visuol.com> Hello, I try to do the following call "getMissionByControlledCar("123")" and I get the following message in my log: "getMissionByControlledCar object not persistent" I passed to the function a parameter (scanexCarId) that doesn't correspond to a row in database. How can I can avoid avoing an exception? td::shared_ptr ODBCarIdenticationServiceImpl::getMissionByControlledCar(std::string scanexCarId) { try { transaction t (m_db->begin ()); std::shared_ptr _mission (m_db->load (scanexCarId)); t.commit (); if(_mission.get() != 0) { return _mission; } else { return nullptr; } } catch (const odb::exception& e) { m_logger->logError("getMissionByControlledCar " + std::string(e.what ())); return nullptr; } } From boris at codesynthesis.com Thu Nov 12 10:19:15 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 12 10:19:12 2015 Subject: [odb-users] Exception : "object not persistent" when trying to load object which does not exist in database In-Reply-To: <4b577fabf112659aeadc19492484b594@visuol.com> References: <4b577fabf112659aeadc19492484b594@visuol.com> Message-ID: Hi Florent, Florent GRATTA writes: > How can I can avoid avoing an exception? > > std::shared_ptr _mission (m_db->load (scanexCarId)); Use find() instead of load(). BTW, this is all discussed in the manual. Boris From boris at codesynthesis.com Thu Nov 12 10:37:43 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 12 10:37:41 2015 Subject: [odb-users] Combining views queries conditions In-Reply-To: <1447336511.152752731@f361.i.mail.ru> References: <1447336511.152752731@f361.i.mail.ru> Message-ID: Hi, ?????? ???????? writes: > #pragma db object > class Object...; > > #pragma db view object(Object) > class View...; > > void f( const query& q ) { > auto res = db->query( q ); > } > > I have tried it and it looks like it works. But is it a hack? Yes, there are cases where this will/might not work. For example, a view could join the same object multiple types (and assign aliases to distinguish between them). If you pass a query condition that is based on the object, it will be ambiguous. However, in simple cases, where you don't assign aliases to objects in views, you can reasonably expect this to work (both queries will use the same, un-aliased table underneath). > #pragma db view object(Object1) object(Object2:...) > class View2...; > > I understand that it is possible to set conditions right in terms of the > View fields, but I want to encapsulate creation subconditions in different > parts of application, and reuse these parts in different places where > different Views are used. Or maybe there exist recommended way (some best > practices) to do this. Hm, I would say, use templates: template auto f (const Q1& q1, const Q2& q2) { return q1 && q2; } using Q = query; db->query (f (Q::Object1::name == "Jonn", Q::Object2::age == 18)); Boris From gratta at visuol.com Thu Nov 12 10:27:08 2015 From: gratta at visuol.com (Florent GRATTA) Date: Thu Nov 12 10:38:23 2015 Subject: [odb-users] Exception : "object not persistent" when trying to load object which does not exist in database In-Reply-To: References: <4b577fabf112659aeadc19492484b594@visuol.com> Message-ID: <965ed686f449773bb7fa50f262d17e84@visuol.com> Hi, Thank's a lot for your answer. I apologize because I have not pay attention on the details in "3.9 Loading Persistent Objects". I have also the "Object not persistent" exception when I try to load an object that is in database but with a null value in relation column (null value is allowed in db, but odn thourght the exception) Please see the code below std::shared_ptr>> ODBCarIdenticationServiceImpl::listControlledCarsByStatus(unsigned char status) { typedef odb::query query; typedef odb::result result; std::shared_ptr>> listControlledCars (new std::vector>()); try { transaction t (m_db->begin ()); result r (m_db->query (query::status <= status && query::lockedByUser == "")); for (result::iterator i (r.begin ()); i != r.end (); ++i) { std::shared_ptr p (i.load()); listControlledCars->push_back(p); } t.commit (); } catch (const odb::exception& e) { m_logger->logError("listControlledCarsByStatus " + std::string(e.what ())); //return nullptr; } return listControlledCars; } Le 12.11.2015 16:19, Boris Kolpackov a ?crit?: > Hi Florent, > > Florent GRATTA writes: > >> How can I can avoid avoing an exception? >> >> std::shared_ptr _mission (m_db->load (scanexCarId)); > > Use find() instead of load(). BTW, this is all discussed in the manual. > > Boris From boris at codesynthesis.com Thu Nov 12 10:56:05 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 12 10:56:02 2015 Subject: [odb-users] Exception : "object not persistent" when trying to load object which does not exist in database In-Reply-To: <965ed686f449773bb7fa50f262d17e84@visuol.com> References: <4b577fabf112659aeadc19492484b594@visuol.com> <965ed686f449773bb7fa50f262d17e84@visuol.com> Message-ID: Hi Florent, Florent GRATTA writes: > I have also the "Object not persistent" exception when I try to load > an object that is in database but with a null value in relation > column (null value is allowed in db, but odn thourght the exception) But is NULL value allowed in ODB? In other words, do you have '#pragma db null' for the data member corresponding to the relation column? Maybe that's what causes the exception to be thourght? Boris From gratta at visuol.com Thu Nov 12 11:04:52 2015 From: gratta at visuol.com (Florent GRATTA) Date: Thu Nov 12 11:13:10 2015 Subject: [odb-users] Exception : "object not persistent" when trying to load object which does not exist in database In-Reply-To: References: <4b577fabf112659aeadc19492484b594@visuol.com> <965ed686f449773bb7fa50f262d17e84@visuol.com> Message-ID: <001c01d11d63$de7a2650$9b6e72f0$@visuol.com> Thank's a lot. You right, no "#pragma db null" is set to the relation. What is the difference between null/not_null and value_null/value_not_null ? Because I tried to specified "#pragma db value_null" on std::string field and the column was created as TEXT NOT NULL Regards, -----Message d'origine----- De?: Boris Kolpackov [mailto:boris@codesynthesis.com] Envoy??: jeudi 12 novembre 2015 16:56 ??: Florent GRATTA Cc?: odb-users@codesynthesis.com Objet?: Re: [odb-users] Exception : "object not persistent" when trying to load object which does not exist in database Hi Florent, Florent GRATTA writes: > I have also the "Object not persistent" exception when I try to load > an object that is in database but with a null value in relation column > (null value is allowed in db, but odn thourght the exception) But is NULL value allowed in ODB? In other words, do you have '#pragma db null' for the data member corresponding to the relation column? Maybe that's what causes the exception to be thourght? Boris From cfdx at list.ru Fri Nov 13 04:08:49 2015 From: cfdx at list.ru (=?UTF-8?B?0JTQsNC90LjQuNC7INCh0LDQstGH0LXQvdC60L4=?=) Date: Fri Nov 13 04:08:58 2015 Subject: =?UTF-8?B?UmVbMl06IFtvZGItdXNlcnNdIENvbWJpbmluZyB2aWV3cyBxdWVyaWVzIGNv?= =?UTF-8?B?bmRpdGlvbnM=?= In-Reply-To: References: <1447336511.152752731@f361.i.mail.ru> Message-ID: <1447405729.696834899@f399.i.mail.ru> Boris, thank you for response and explanation. As I understand the limitation is absence of aliases objects in the target view and it?s not a hack in this case. Thank you. Daniil Savchenko ? ?????????, ?????? ?. ???????? cfdx@list.ru From markus at markusklemm.net Fri Nov 13 06:04:02 2015 From: markus at markusklemm.net (Markus Klemm) Date: Fri Nov 13 06:04:18 2015 Subject: [odb-users] boost::optional query::in_range compile error Message-ID: Hi there again, is it possible that query::in_range when used with boost::optional data members? (in a container of course and used with their iterator). Using ODB 2.4.0 and VS 2013, let the example object have the member: (full example syntax highlighted here: https://gist.github.com/Superlokkus/7fc1545f33d10d2caae8 ) boost::optional is_ok; and a translation unit using std::vector values; odb::query::is_ok.in_range(values.cbegin(), values.cend()); one will get the compile error: error C2665: 'odb::mssql::val_bind::val_bind' : none of the 2 overloads could convert all the argument types Defining the member as non-optional worked as usual. Regards (and thank you in advance to Boris probably ;-) ) Markus Klemm From boris at codesynthesis.com Fri Nov 13 09:46:26 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Nov 13 09:46:22 2015 Subject: [odb-users] Exception : "object not persistent" when trying to load object which does not exist in database In-Reply-To: <001c01d11d63$de7a2650$9b6e72f0$@visuol.com> References: <4b577fabf112659aeadc19492484b594@visuol.com> <965ed686f449773bb7fa50f262d17e84@visuol.com> <001c01d11d63$de7a2650$9b6e72f0$@visuol.com> Message-ID: Florent GRATTA writes: > What is the difference between null/not_null and value_null/value_not_null ? It is all in the manual. See Section 14.4.28. Boris From boris at codesynthesis.com Mon Nov 16 08:02:01 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Nov 16 08:01:55 2015 Subject: [odb-users] boost::optional query::in_range compile error In-Reply-To: References: Message-ID: Hi Markus, Markus Klemm writes: > boost::optional is_ok; > > std::vector values; > odb::query::is_ok.in_range(values.cbegin(), values.cend()); For query purposes, ODB "strips" the wrapper (boost::optional in this case). So std::vector is what will work. This is actually what you most likely want anyway since SQL 'IN' is not going to do anything sensible if you pass it a NULL value. Boris From sverre at awion.dk Tue Nov 17 11:21:42 2015 From: sverre at awion.dk (Sverre Eplov) Date: Tue Nov 17 11:21:55 2015 Subject: [odb-users] Looking for the most efficent way to load a object hiearchy Message-ID: <90476E25-9238-4B4C-B6A9-FC7E73683897@awion.dk> Hi, I am currently playing ?getting to know ODB?, and I think I?ve worked out how to do most of what I want to do. There is one thing I can?t really work out - what is the most efficient way to load into memory a object hierachy, where some objects have a one-to-many relationship to other objects. So if I have tables A, B, C and D, where - A have a one-to-one relationship to B - B have a one-to-many relationship to C - C have a one-to-many relationship to D Example instance graph: A - B + C1 + C2 + C3 + D1 + D2 I would now like to load a number of A?s (say, about 200) with full object graph as well. How can this be done most efficiently ? I can?t see a way using object views, as they do not allow the objects they load to have collections in them. I don?t exactly need actual code for this, more a general descriptoin of the best way to achieve loading of a complete hierachy. Lazy loading won?t do, as I need to serialize the whole bazinga and ship it over a wire to a client app. (BTW, the background is that in order to test out ODB, I?m reconstructing in C++ a solution I?ve done for a customer in C#, where I have used the Entity Framework as ORM and WPF frontend. I?m redoing in C++ using ODB and QT. ) Best Regards, /Sverre Eplov From odb at a-cunningham.com Tue Nov 17 12:28:36 2015 From: odb at a-cunningham.com (Andrew Cunningham) Date: Tue Nov 17 12:28:43 2015 Subject: [odb-users] Re: odb-users Digest, Vol 63, Issue 7 In-Reply-To: <201511171700.tAHH0g4a013614@codesynthesis.com> References: <201511171700.tAHH0g4a013614@codesynthesis.com> Message-ID: Hi Sverre, The brilliance of ODB is you just code your relationships using C++ and familiar STL collections and you are 'done'. #pragma db object pointer(std::shared) class D { ... } #pragma db object pointer(std::shared) class C { ... std:: list< std::shared_ptr > dlist_; } #pragma db object pointer(std::shared) class B { ... std:: list< std::shared_ptr > clist_; } #pragma db object pointer(std::shared) class A { .... std::shared_ptr b_ } When you load your object ( or list of objects A's) from the database, the complete hierarchy is loaded. > Hi, > > I am currently playing ?getting to know ODB?, and I think I?ve worked out > how to do most of what I want to do. > > There is one thing I can?t really work out - what is the most efficient > way to load into memory a object hierachy, where some objects have a > one-to-many relationship to other objects. > > > So if I have tables A, B, C and D, where > > - A have a one-to-one relationship to B > - B have a one-to-many relationship to C > - C have a one-to-many relationship to D > > Example instance graph: > > A - B + C1 > + C2 > + C3 + D1 > + D2 > > > I would now like to load a number of A?s (say, about 200) with full object > graph as well. How can this be done most efficiently ? I can?t see a way > using object views, as they do not allow the objects they load to have > collections in them. > > I don?t exactly need actual code for this, more a general descriptoin of > the best way to achieve loading of a complete hierachy. Lazy loading won?t > do, as I need to serialize the whole bazinga and ship it over a wire to a > client app. > > > (BTW, the background is that in order to test out ODB, I?m reconstructing > in C++ a solution I?ve done for a customer in C#, where I have used the > Entity Framework as ORM and WPF frontend. I?m redoing in C++ using ODB and > QT. ) > > From boris at codesynthesis.com Wed Nov 18 10:08:15 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Nov 18 10:08:21 2015 Subject: [odb-users] Looking for the most efficent way to load a object hiearchy In-Reply-To: <90476E25-9238-4B4C-B6A9-FC7E73683897@awion.dk> References: <90476E25-9238-4B4C-B6A9-FC7E73683897@awion.dk> Message-ID: Hi Sverre, Sverre Eplov writes: > A - B + C1 > + C2 > + C3 + D1 > + D2 > > I would now like to load a number of A?s (say, about 200) with full object > graph as well. How can this be done most efficiently? When it comes to to-many relationships, it is hard to achieve absolute optimum performance without a significant amount of manual work. Specifically, joining everything together (i.e., the object loading view way) will not necessarily result in better performance since you will be loading multiple copies of C, B, and A objects. So the recommended way is to start with essentially what Andrew suggested: let ODB load everything in the default way. BUT (and this can make a huge difference to the performance), make sure that all your objects have session support enabled and you have an active session (object cache). This means that if you have the same B/C/D objects pointed-to from multiple places, they will only be loaded from the database once. Only if/when this results in unacceptable performance, should you think about optimizing it. And the way to do it is essentially load everything (using object loading views) backwards, starting from all the D's, then C's, then B's, and finally A's. Boris From meike.talbach at women-at-work.org Wed Nov 18 10:55:24 2015 From: meike.talbach at women-at-work.org (meike.talbach@women-at-work.org) Date: Wed Nov 18 10:55:32 2015 Subject: [odb-users] Prepared statements alreadys exists Message-ID: Dear all, ? I'm trying to use ODB in a new project with a PostgreSQL database. I basically have this code: ? void test(boost::posix_time::ptime from, boost::posix_time::ptime to) { std::shared_ptr db = ...; odb::transaction txn(db->begin()); database::HistoryQuery qryHistories(database::HistoryQuery::device == dbDevice->GetId()); qryHistories += " AND (" + (database::HistoryQuery::dateRecorded >= from) + ")"; qryHistories += " AND (" + (database::HistoryQuery::dateRecorded <= cmp) + ")"; odb::result result(db->query(query)); if(!result.empty()) result.begin().load(); } I then call test with invalid ptimes which is expected to fail. However my second call should succeed, but in fact fails too. Something like this: void testWrapper() { try { test(boost::posix_time::ptime(boost::posix_time::special_values::not_a_date_time), boost::posix_time::ptime(boost::posix_time::special_values::not_a_date_time)) } catch(const std::exception &ex) { std::cout << "fail" << std::endl; } try { test(boost::posix_time::ptime(boost::gregorian::date(2015, 11, 18)), boost::posix_time::ptime(boost::gregorian::date(2015, 11, 18))); } catch(const std::exception &ex) { std::cout << "fail" << std::endl; } } Postgres logs this error: Oct 24 15:12:53 test1 postgres[52801]: [3-1] ERROR: timestamp out of range Oct 24 15:12:53 test1 postgres[52801]: [3-2] STATEMENT: SELECT "test_history"."id", "test_history"."deviceid", "test_history"."type", "test_history"."latitude", "test_history"."longitude", "test_history"."address", "test_history"."daterecorded" FROM "test_history" WHERE "test_history"."deviceid" = $1 AND ("test_history"."daterecorded" >= $2) AND ("test_history"."daterecorded" <= $3) Oct 24 15:12:53 test1 postgres[52801]: [4-1] ERROR: current transaction is aborted, commands ignored until end of transaction block Oct 24 15:12:53 test1 postgres[52801]: [4-2] STATEMENT: deallocate "mc_test_database_History_query" Oct 24 15:54:19 test1 postgres[52801]: [5-1] ERROR: prepared statement "mc_test_database_History_query" already exists Oct 24 15:54:19 test1 postgres[52801]: [5-2] STATEMENT: SELECT "test_history"."id", "test_history"."deviceid", "test_history"."type", "test_history"."latitude", "test_history"."longitude", "test_history"."address", "test_history"."daterecorded" FROM "test_history" WHERE "test_history"."deviceid" = $1 ORDER BY "test_history"."daterecorded" ASC LIMIT 1 It seems that the prepared statement from the first call is not deallocated. Then the second call tries to recreate it and fails. Any ideas how to solve this ? Thank you Meike From boris at codesynthesis.com Thu Nov 19 09:07:32 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 19 09:07:35 2015 Subject: [odb-users] Prepared statements alreadys exists In-Reply-To: References: Message-ID: Hi Meike, meike.talbach@women-at-work.org writes: > Postgres logs this error: > > Oct 24 15:12:53 test1 postgres[52801]: [3-1] ERROR: timestamp out of range > Oct 24 15:12:53 test1 postgres[52801]: [3-2] STATEMENT: SELECT "test_history"."id", "test_history"."deviceid", "test_history"."type", "test_history"."latitude", "test_history"."longitude", "test_history"."address", "test_history"."daterecorded" FROM "test_history" WHERE "test_history"."deviceid" = $1 AND ("test_history"."daterecorded" >= $2) AND ("test_history"."daterecorded" <= $3) > Oct 24 15:12:53 test1 postgres[52801]: [4-1] ERROR: current transaction is aborted, commands ignored until end of transaction block > Oct 24 15:12:53 test1 postgres[52801]: [4-2] STATEMENT: deallocate "mc_test_database_History_query" > Oct 24 15:54:19 test1 postgres[52801]: [5-1] ERROR: prepared statement "mc_test_database_History_query" already exists > Oct 24 15:54:19 test1 postgres[52801]: [5-2] STATEMENT: SELECT "test_history"."id", "test_history"."deviceid", "test_history"."type", "test_history"."latitude", "test_history"."longitude", "test_history"."address", "test_history"."daterecorded" FROM "test_history" WHERE "test_history"."deviceid" = $1 ORDER BY "test_history"."daterecorded" ASC LIMIT 1 > > It seems that the prepared statement from the first call is not deallocated. > Then the second call tries to recreate it and fails. Thanks for the detailed information, always appreciated! I think I now understand what's going on here. Below is a comment for the fix: // When we try to execute an invalid statement, PG "poisons" the // transaction (those "current transaction is aborted, commands // ignored until end of transaction block" messages in the log). // This includes prepared statement deallocations (quite a stupid // decision, if you ask me). // // So what can happen in this situation is the deallocation fails // but we ignore it because we are already unwinding the stack // (i.e., the prepared statement execution has failed). Next the // user fixes things (e.g., passes valid parameters) and tries to // re-execute the same query. But since we have failed to deallocate // the statement, we now cannot re-prepare it; the name is already // in use. And here is the fix itself: http://scm.codesynthesis.com/?p=odb/libodb-pgsql.git;a=commit;h=231d0222fb75c448321677d258ed59da98eec7ee Could you apply this patch to your libodb-pgsql and verify that your code now works as expected? Thanks, Boris From meike.talbach at women-at-work.org Fri Nov 20 04:04:14 2015 From: meike.talbach at women-at-work.org (meike.talbach@women-at-work.org) Date: Fri Nov 20 04:04:22 2015 Subject: Aw: Re: [odb-users] Prepared statements alreadys exists In-Reply-To: References: , Message-ID: Hi Boris, I tried your patch and it seems to work correctly now. Will let you know if I experience any further problems. Thank you for your support Meike From zeyangtao1020 at gmail.com Fri Nov 20 21:43:53 2015 From: zeyangtao1020 at gmail.com (Zeyang Tao) Date: Mon Nov 23 03:21:10 2015 Subject: [odb-users] Nested vector Message-ID: Hi, Recently we had a slowness issue in ODB, the thing is ODB doesn't support nested vector, i.e vector of vector, so in order to store that kind of data structure, we have to use the following hacky way, struct node{ int x, y; double value; } class parameter{ ... ... vector cnt; } basically the cnt is a matrix and we use x and y as the index, each single node will be a record in the sub table, sometimes the matrix is large and one update will write N^2 records to the subtable, plus this update is very frequent, so we saw 2-3 seconds delay constantly I wonder if ODB has some features to optimize this or do you have some suggestions ? Thanks, Zey From boris at codesynthesis.com Mon Nov 23 08:32:16 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Nov 23 08:32:16 2015 Subject: [odb-users] Nested vector In-Reply-To: References: Message-ID: Hi, Zeyang Tao writes: > Recently we had a slowness issue in ODB, the thing is ODB doesn't support > nested vector, i.e vector of vector, so in order to store that kind of data > structure, we have to use the following hacky way, > > struct node{ > int x, y; > double value; > } > > class parameter{ > ... > ... > vector cnt; > } One thing you can improve on here is to use the vector's index for 'x', e.g., struct node { size_t y; double value; }; vector cnt; Or, if you want to keep x and y, you can mark the cnt member as unordered (Section 14.4.19). > basically the cnt is a matrix and we use x and y as the index, each single > node will be a record in the sub table, sometimes the matrix is large and > one update will write N^2 records to the subtable, plus this update is very > frequent, so we saw 2-3 seconds delay constantly I think the first thing you should try is the change-tracking vector (Section 5.4). Unless you always modify most of your elements, I bet it will solve your problem. Also, there could be some database system-specific ways to optimize this (e.g., some databases support arrays of simple types). But since you didn't mention which database you are using (always are good idea), I can't recommend anything specific in this area. Boris From zeyangtao1020 at gmail.com Mon Nov 23 22:33:59 2015 From: zeyangtao1020 at gmail.com (Zeyang Tao) Date: Tue Nov 24 05:16:09 2015 Subject: [odb-users] Nested vector In-Reply-To: References: Message-ID: Hi Boris, Thanks for your reply. I am using postgresql. Yes, I used odb::vector and only updated the elements which have been changed. It improved the performance a lot but we can still see the slowness issue accidentally. If I have pre knowledge about the matrix size, I think I can get rid of X and Y, do you think that will make a big difference ? Thanks 2015-11-23 7:32 GMT-06:00 Boris Kolpackov : > Hi, > > Zeyang Tao writes: > > > Recently we had a slowness issue in ODB, the thing is ODB doesn't support > > nested vector, i.e vector of vector, so in order to store that kind of > data > > structure, we have to use the following hacky way, > > > > struct node{ > > int x, y; > > double value; > > } > > > > class parameter{ > > ... > > ... > > vector cnt; > > } > > One thing you can improve on here is to use the vector's index for 'x', > e.g., > > struct node > { > size_t y; > double value; > }; > > vector cnt; > > Or, if you want to keep x and y, you can mark the cnt member as > unordered (Section 14.4.19). > > > > basically the cnt is a matrix and we use x and y as the index, each > single > > node will be a record in the sub table, sometimes the matrix is large and > > one update will write N^2 records to the subtable, plus this update is > very > > frequent, so we saw 2-3 seconds delay constantly > > I think the first thing you should try is the change-tracking vector > (Section 5.4). Unless you always modify most of your elements, I bet > it will solve your problem. > > Also, there could be some database system-specific ways to optimize > this (e.g., some databases support arrays of simple types). But since > you didn't mention which database you are using (always are good idea), > I can't recommend anything specific in this area. > > Boris > From hmojtaba89 at gmail.com Tue Nov 24 08:21:52 2015 From: hmojtaba89 at gmail.com (Mojtaba Hosseini) Date: Tue Nov 24 08:40:36 2015 Subject: [odb-users] Compiling examples on Eclipse Message-ID: (Eclipse on linux ubuntu 14.04) i'm getting this error: /odb/include/odb/mysql/version.hxx:16:35: fatal error: mysql/mysql_version.h: No such file or directory what is the problem!!? there is actually no mysql_verson.h file in my Ubuntu. I'v installed everything right! the examples can be compiled with terminal commands, but not on eclipse! From boris at codesynthesis.com Tue Nov 24 09:09:18 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Nov 24 09:09:16 2015 Subject: [odb-users] Nested vector In-Reply-To: References: Message-ID: Hi, Zeyang Tao writes: > Yes, I used odb::vector and only updated the elements which have been > changed. It improved the performance a lot but we can still see the > slowness issue accidentally. > > If I have pre knowledge about the matrix size, I think I can get rid of X > and Y, do you think that will make a big difference? I don't think so. I believe (though I can very well be wrong), the issue is not the size of the data but the number of statements that has to be executed. > I am using postgresql. Well, in this case, you can use PG array types. That is, map std::vector to DOUBLE PRECISION[], then use it as an element of odb::vector<> (so each row will only be updated if some of its elements have changed). For more information on how to map PG array types, see "Extended Database to C++ Type Mapping in ODB": http://www.codesynthesis.com/~boris/blog/2012/07/18/custom-database-to-cxx-type-mapping-in-odb/ Plus, the pgsql/custom test in the odb-tests package has a working mapping for INTEGER[]. Also, would appreciate it if you let us whether this helps with your performance issue. Boris From konstantin.tarovik at ab-soft.net Tue Nov 24 19:24:18 2015 From: konstantin.tarovik at ab-soft.net (Konstantin Tarovik) Date: Wed Nov 25 05:56:19 2015 Subject: [odb-users] ODB custom session notifications on transaction rollback Message-ID: <5654FFB2.1080305@ab-soft.net> Hello, I have an issue while using transactions in conjunction with custom session notifications. Please look at the following three lines: odb::transaction t(_db->begin()); _db->persist(c); t.commit(); During persist call (even before commit is called) I get the notification inside custom session telling that the element has been stored to the DB cache (actually I receive two callbacks in a row _cache_persist and _cache_insert). This is not perfect behavior IMO but it's ok. The real issue arises when I rollback the transaction: odb::transaction t(_db->begin()); _db->persist(c); t.rollback(); I still receive the notification inside persist call telling that the element has been added. But when I rollback the transaction _cache_erase notification is not being called. Taking into account that I use notifications to make data consistent between the DB and UI this behavior causes data inconsistency. Am I missing something? Thank you. -- Best regards, Konstantin Tarovik From andrew at a-cunningham.com Tue Nov 24 21:57:57 2015 From: andrew at a-cunningham.com (Andrew Cunningham) Date: Wed Nov 25 05:56:19 2015 Subject: [odb-users] Re: Nested vector Message-ID: Another approach that I have used very effectively with SQLite is to use a BLOB of vector to store the array data. I saw a dramatic performance boost in my code. Effectively you load/save the whole BLOB - it's data is opaque to SQL. You may need to be aware of any alignment issues when casting a node * to unsigned char *. And I'm not sure of BLOB size limits in postgres. From boris at codesynthesis.com Wed Nov 25 06:40:24 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Nov 25 06:40:23 2015 Subject: [odb-users] ODB custom session notifications on transaction rollback In-Reply-To: <5654FFB2.1080305@ab-soft.net> References: <5654FFB2.1080305@ab-soft.net> Message-ID: Hi Konstantin, Konstantin Tarovik writes: > odb::transaction t(_db->begin()); > > _db->persist(c); > > t.commit(); > > During persist call (even before commit is called) I get the > notification inside custom session telling that the element has been > stored to the DB cache (actually I receive two callbacks in a row > _cache_persist and _cache_insert). This is not perfect behavior IMO > but it's ok. The real issue arises when I rollback the transaction: > > odb::transaction t(_db->begin()); > > _db->persist(c); > > t.rollback(); > > I still receive the notification inside persist call telling that > the element has been added. But when I rollback the transaction > _cache_erase notification is not being called. Taking into account > that I use notifications to make data consistent between the DB and > UI this behavior causes data inconsistency. The custom session interface is intentionally low-level in order to allow various session/cache semantics. For example, some session implementation may only be intended to live for as long as the transaction. For such a session any kind of cross-transaction state tracking would be unnecessary. But if you do want the semantics that you have described, then you will need to implement it on top of this low-level session interface with the help of transaction callbacks (Section 15.1, "Transaction Callbacks"). The idea is to first insert the objects "tentatively" (e.g., by setting a flag). Then, in the transaction callback, depending on whether it is committed or rolled back, you either "finalize" such objects (e.g., by clearing the flag) or remove them from the session. In fact, the change-tracking session implementation in the common/session/custom test in the odb-tests package does something very similar: it clears the change flag on an object entry only if and when the transaction is committed. Boris From finjulhich at gmail.com Fri Nov 27 19:09:24 2015 From: finjulhich at gmail.com (MM) Date: Fri Nov 27 19:09:33 2015 Subject: [odb-users] Re: libcutl repo not exported In-Reply-To: References: Message-ID: On 29 September 2015 at 15:50, Boris Kolpackov wrote: > Hi, > > MM writes: > > > fatal: remote error: access denied or repository not exported: > > /odb/libcutl/libcutl.git > > I wonder when will you learn to tell what you have actually done > (i.e. the *exact* commands that you ran) that triggered the error? > In fact, enough is enough. From now on, I will be ignoring your > questions that don't include this information. > > In any case, both: > > git clone git://scm.codesynthesis.com/libcutl/libcutl.git > git clone http://scm.codesynthesis.com/libcutl/libcutl.git > > As suggested on: > > http://scm.codesynthesis.com/?p=libcutl/libcutl.git;a=summary > > Work for me. > > > > I now follow INSTALL-GIT and just do a make inside the odb directory. It > > seems this file options.hxx is not generated, even > > > > odb/odb/odb.cxx:38:27: fatal error: odb/options.hxx: No such file or > > directory > > When you ran make for the first time, you should have been asked > a bunch of questions, including whether you would like to use the > development build of cli (you should answer 'no') under Fedora 23, in my dir ~/srcs/codesynthesis, I've cloned the following: build cli libodb libodb-boost libodb-sqlite odb odb-etc by using this command once for each directory: git clone git://scm.codesynthesis.com/ After having built and installed build and cli, with cli installed in /usr/local/bin/cli, and now I am inside odb, I run make for the first time: Configuring external dependency on 'cli' for 'odb'. Would you like to configure dependency on the installed version of 'cli' as opposed to the development build? [y]: and where cli > is (you should answer '/usr/local/bin/cli'). After that, the > build process should generate options.?xx without problems. > If I press Enter which therefore means 'y', it doesn't ask me for where is 'cli'. I finish with all the questions (I take defaults for all), and then the above error happens. If I answer 'n', it asks me for src_root [] for cli and out_root [/usr/local/bin] From boris at codesynthesis.com Mon Nov 30 09:34:21 2015 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Nov 30 09:34:23 2015 Subject: [odb-users] Re: libcutl repo not exported In-Reply-To: References: Message-ID: Hi, MM writes: > If I press Enter which therefore means 'y', it doesn't ask me for where is > 'cli'. I finish with all the questions (I take defaults for all), and then > the above error happens. Ok, I managed to reproduce and fix this. Can you pull and try again? And thanks for sticking around to figure this out! Boris From finjulhich at gmail.com Mon Nov 30 10:59:52 2015 From: finjulhich at gmail.com (MM) Date: Mon Nov 30 11:00:01 2015 Subject: [odb-users] Re: libcutl repo not exported In-Reply-To: References: Message-ID: On 30 November 2015 at 14:34, Boris Kolpackov wrote: > Hi, > > MM writes: > > > If I press Enter which therefore means 'y', it doesn't ask me for where > is > > 'cli'. I finish with all the questions (I take defaults for all), and > then > > the above error happens. > > Ok, I managed to reproduce and fix this. Can you pull and try again? > And thanks for sticking around to figure this out! > > Boris > > I messed up my install quite a bit. I'll restart from scratch now. Would you be able to bear with me step by step because I found it extremely difficult:-) Starting from scratch: I have fedora 23 and have no odb rpm installed, only libcutl: libcutl-1.9.0-5.fc23.x86_64 (/usr/lib64/libcutl-1.9.so /usr/share/doc/libcutl ) libcutl-devel-1.9.0-5.fc23.x86_64 (files under /usr/include/cutl, /usr/lib64/libcutl.so and /usr/lib64/pkgconfig/libcutl.pc) and bash-4.3.42-1.fc23.x86_64 make-4.0-5.1.fc23.x86_64 m4-1.4.17-8.fc23.x86_64 sed-4.2.2-11.fc23.x86_64 autogen-5.18.6-1.fc23.x86_64 libtool-2.4.6-5.fc23.x86_64 shtool-2.0.8-16.fc23.noarch automake-1.15-4.fc23.noarch autoconf-2.69-21.fc23.noarch tofrodos-1.7.13-5.fc23.x86_64 in ~/srcs/codesynthesis, I do: git clone http://scm.codesynthesis.com/xxxx for the following: 1. build/build.git 2. cli/cli.git 3. odb/odb.git 4. odb/libodb.git 5. odb/libodb-sqlite.git 6. odb/libodb-boost.git I assume the deps are such that 1 and 2 have to be build / installed, and then proceed with 3 onwards. What do I do next? Please only 1 step at a time :-) Thanks MM