From finjulhich at gmail.com Fri Sep 1 05:18:38 2023 From: finjulhich at gmail.com (MM) Date: Fri Sep 1 05:09:55 2023 Subject: [odb-users] feature request: allow callback to specify out of class function Message-ID: I believe at the moment https://www.codesynthesis.com/products/odb/doc/manual.xhtml#14.1.7 "callback" argument needs to be an in-class member function. Please allow it to be an out-of-class function. The reason is I keep everything related to odb in a separate part of the sources. Rds, From boris at codesynthesis.com Fri Sep 1 10:25:20 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Sep 1 10:16:19 2023 Subject: [odb-users] odb abstract classes/tables cannot be pointed to In-Reply-To: References: Message-ID: MM writes: > class Base1 { > Base2* b2; > > const b2& get_b2() const; > void set_b2(Base2&); > }; > #pragma db object(Base1) abstract definition table("B1") > #pragma db member(Base1::b2) not_null column("b2")\ > get(&this.get_b2()) set(this.set_b2(*(?))) > > #pragma db object(Base2) abstract definition table("B2") > class Base2 {...}; > > class D11 : public Base1 {}; class D12 : public Base1 {}; > class D21 : public Base2 {}; class D22 : public Base2 {}; > > odb compiler fails with > "pointed-to class 'Base1' is abstract" > > The error makes sense. How can I go about solving this? > all Dxx derived tables are in the database, and will have additional > specific columns. > However the association 1 to 1 Base1 => Base2 remains true An ODB object relationship is mapped to a foreign key in the SQL database and a foreign key can only refer to one table. So based on that you either (1) need to make D2X use the same table or (2) not use object relationships. The way to achieve (1) in ODB is to use polymorphic objects: https://www.codesynthesis.com/products/odb/doc/manual.xhtml#8.2 The way to achieve (2) is to use a value (e.g., an object id) rather that a pointer for Base1::b2 and then do the pointer-to object loading yourself. From finjulhich at gmail.com Sun Sep 3 14:13:38 2023 From: finjulhich at gmail.com (MM) Date: Sun Sep 3 14:04:55 2023 Subject: [odb-users] after sqlite, choosing between postgresql and mysql Message-ID: Hello, We are expanding our state in that we need to have a client server database system to allow for concurrent read and write from different nodes. odb is finally paying off:-) This is a hard question but let me describe our use case 1. small number of concurrent users (under dozen) 2. 3 databases in use, some read only, some read and write 3. 2 dozen tables in each database 4. Nbr of rows in each table ranges from 10 to 100 000 max 5. Nbr of columns from 3/4 to 2 dozen columns, but only few tables 6. concurrency: most of the transactions are read. 10% perhaps or less are writes. 7. There may be a desire to do a cross database "transaction", i.e. either both databases are written successfully to or the transaction reverts on both. That is what I can think of at the moment. Any advice? MM From finjulhich at gmail.com Sun Sep 3 14:32:58 2023 From: finjulhich at gmail.com (MM) Date: Sun Sep 3 14:24:15 2023 Subject: [odb-users] update() doesn't offer a constT& signature Message-ID: Hello I get compile errors when I try to do void f(const & myobj) { ... db.update(myobj); ... } There are a number of overloads in database.hxx but none of them takes const T& alone. Rds, From lloydkl.tech at gmail.com Mon Sep 4 02:37:51 2023 From: lloydkl.tech at gmail.com (Lloyd) Date: Mon Sep 4 02:30:37 2023 Subject: [odb-users] Foreign key - Query executes multiple times on parent table Message-ID: We are using ODB (2.4.0) with sqlite-3 (3.41.2) We have two tables connected using foreign key relationship. The child table has 3 columns and 0.25 million records. The parent table has 4 columns and 2 records. In our C++ program, I have given a "select *" query without any condition on the child table and the results are copied in an iterative loop. When tracing the query execution using ODB tracer, we observe that In each iteration a "select query" is executed on the parent table also (Isn't it unnecessary?). This makes the query execution very slow (around 45 minutes). The table structure and the code fragment is given below. How can I avoid the "select query" execution on the parent table? CREATE TABLE "Parent" ( "Id" INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, "F" TEXT NOT NULL, "S" INTEGER NOT NULL, "TVS" TEXT NOT NULL); CREATE TABLE "Child" ( "Id" INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, "N" TEXT NOT NULL, "IId" INTEGER NOT NULL, CONSTRAINT "IId_fk" FOREIGN KEY("IId") REFERENCES "Parent"("Id") ON DELETE CASCADE DEFERRABLE INITIALLY DEFERRED ); Following is the Implementation of the SQLite-specific Tracer #include #include #include #include class sqlitetracer : public odb::sqlite::tracer { virtual void prepare(odb::sqlite::connection& c, const odb::sqlite::statement& s) override { cerr << c.database().name()/*.db()*/ << ": PREPARE " << s.text() /*<< " AS " << s.text()*/; } virtual void execute(odb::sqlite::connection& c, const odb::sqlite::statement& s) override { cerr << c.database().name() << ": EXECUTE " << s.text(); } virtual void execute(odb::sqlite::connection& c, const char* statement) override { cerr << c.database().name() << ": " << statement; } virtual void deallocate(odb::sqlite::connection& c, const odb::sqlite::statement& s) override { cerr << c.database().name() << ": DEALLOCATE " << s.text(); } }; The above tracer returns 2 SQL statements : 1. SELECT "Child"."Id", "Child"."N", "Child"."IId" WHERE "Child"."Id" = 2; 2. SELECT "Parent"."Id", "Parent"."F", "Parent"."S", "Parent"."TVS" WHERE "Parent"."Id"=1; The first query is executed once and the second query is executed for 0.25 million times. The following is the C++ code for reading the data typedef unsigned long long C_UINT64; PRAGMA_DB(object) class Parent { private: Parent(){} friend class odb::access; PRAGMA_DB(id auto) C_UINT64 Id; String F; CC_UINT64 S; std::string TVS; }; PRAGMA_DB(object) class Child { private: Child(){} friend class odb::access; PRAGMA_DB(id auto) CC_UINT64 Id; CC_UINT64 IId; std::string N; PRAGMA_DB(not_null) PRAGMA_DB(on_delete(cascade)) PRAGMA_DB(index) std::shared_ptrIId; public: ... inline C_UINT64 GetIId() { return IId; } }; typedef odb::result result; typedef odb::query query; transaction t(DBPtr.begin()); result r = DBPtr.query( query::ID == id); for (result::iterator fdt(r.begin());fdt != r.end(); ++fdt) { fdt->GetIId();//Too slow. Executes a select query on parent } t.commit(); From Visa.Karthick at irisi.com.au Sun Sep 3 23:41:34 2023 From: Visa.Karthick at irisi.com.au (Visa Karthick) Date: Mon Sep 4 05:10:33 2023 Subject: [odb-users] How to run a select query to a DB from C++? Message-ID: I am troubleshooting a program and need some assistance. Below is the code. void initialiseDB(std::unique_ptr& db) { try { odb::transaction t2(db->begin()); // Find function should be invoked here auto tankType = std::make_shared("Tank"); db->persist(tankType.get()); // This inserts a new record with value Tank in the EntityType table. t2.commit(); } catch (const odb::exception &e) { std::cerr << e.what() << std::endl; return; } } The above code works fine for inserting records in to a Sqlite table. The problem is it keeps adding the same row again and again When I run the program. I need to insert records only when the table is empty. There is another function for running a select query in the database. It is invoked by the below code. template inline typename object_traits::pointer_type database:: find (const typename object_traits::id_type& id) { return find_ (id); } template inline bool database:: find (const typename object_traits::id_type& id, T& obj) { return find_ (id, obj); } I need to invoke the find function first, and only if it doesnt return any result persist function should be called. I am unsure what parameters to be passed from my current scope. Could you please advise? I am not sure what parameters to be passed. Visa Karthick | Electrical Engineer | Victoria IRIS Integration Pty Ltd T:+61 1300 240 036 | M : +61 405918505 | visa.karthick@IRISI.com.au From boris at codesynthesis.com Mon Sep 4 05:30:48 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Sep 4 05:21:46 2023 Subject: [odb-users] How to run a select query to a DB from C++? In-Reply-To: References: Message-ID: Visa Karthick writes: > odb::transaction t2(db->begin()); > // Find function should be invoked here > auto tankType = std::make_shared("Tank"); > db->persist(tankType.get()); // This inserts a new record with value Tank in the EntityType table. > t2.commit(); > > The above code works fine for inserting records in to a Sqlite table. > The problem is it keeps adding the same row again and again When I run > the program. I need to insert records only when the table is empty. There > is another function for running a select query in the database. It is > invoked by the below code. > > template > inline typename object_traits::pointer_type database:: > find (const typename object_traits::id_type& id) > { > return find_ (id); > } > > I need to invoke the find function first, and only if it doesnt return > any result persist function should be called. I am unsure what parameters > to be passed from my current scope. Could you please advise? The usage of the database::find() function is discussed in Section 3.9, "Loading Persistent Objects": https://www.codesynthesis.com/products/odb/doc/manual.xhtml#3.9 Specifically, the argument to find() is the object id (i.e., the value that corresponds to data member marked with `#pragma db id`). Since you didn't provide the definition for the EntityType class, it's unclear whether "Tank" is the object id. If it is, then that's what you would pass to find(). If it's not, then you will need to run a query to determine if an object a with non-id data member "Tank" is already in the database. For that see Section 2.5, "Querying the Database for Objects": https://www.codesynthesis.com/products/odb/doc/manual.xhtml#2.5 From boris at codesynthesis.com Mon Sep 4 05:39:46 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Sep 4 05:30:43 2023 Subject: [odb-users] Foreign key - Query executes multiple times on parent table In-Reply-To: References: Message-ID: Lloyd writes: > We have two tables connected using foreign key relationship[:] > > CREATE TABLE "Parent" ( > "Id" INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, > "F" TEXT NOT NULL, > "S" INTEGER NOT NULL, > "TVS" TEXT NOT NULL); > > CREATE TABLE "Child" ( > "Id" INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, > "N" TEXT NOT NULL, > "IId" INTEGER NOT NULL, > CONSTRAINT "IId_fk" FOREIGN KEY("IId") REFERENCES "Parent"("Id") ON DELETE > CASCADE DEFERRABLE INITIALLY DEFERRED > ); > > PRAGMA_DB(object) > class Parent > { > private: > Parent(){} > friend class odb::access; > PRAGMA_DB(id auto) > C_UINT64 Id; > String F; > CC_UINT64 S; > std::string TVS; > }; > > PRAGMA_DB(object) > class Child > { > private: > Child(){} > friend class odb::access; > PRAGMA_DB(id auto) > CC_UINT64 Id; > CC_UINT64 IId; > std::string N; > > PRAGMA_DB(not_null) > PRAGMA_DB(on_delete(cascade)) > PRAGMA_DB(index) > std::shared_ptrIId; > public: > ... > inline C_UINT64 GetIId() { return IId; } > }; > > > When tracing the query execution using ODB tracer, we observe that In > each iteration a "select query" is executed on the parent table also > (Isn't it unnecessary?). The default semantics of loading an object in ODB is to also load all its relationships (Child::IId in this case). There are various ways to avoid this (lazy pointers, sections, etc) but they don't really apply to your case (see below). > How can I avoid the "select query" execution on the parent table? > > result r = DBPtr.query( query::ID == id); > for (result::iterator fdt(r.begin());fdt != r.end(); ++fdt) > { > fdt->GetIId();//Too slow. Executes a select query on parent > } You cannot really avoid touching both tables because you need information from both. But you can make it a single query using a view: https://www.codesynthesis.com/products/odb/doc/manual.xhtml#10.1 From boris at codesynthesis.com Mon Sep 4 05:57:21 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Sep 4 05:48:19 2023 Subject: [odb-users] update() doesn't offer a constT& signature In-Reply-To: References: Message-ID: MM writes: > I get compile errors when I try to do > > void f(const & myobj) { > ... > db.update(myobj); > ... > } > > There are a number of overloads in database.hxx but none of them takes > const T& alone. The const version of database::update() is available unless update() would need to modify the object during update. This happens, for example, if the object uses optimistic concurrency and update() has to increment the version. It's impossible to say what exactly causes this in your case without seeing the definition of MyType as well as the error. In fact, always providing the exact diagnostics you get as well as all the relevant code fragments is strongly recommended. From finjulhich at gmail.com Tue Sep 5 12:50:44 2023 From: finjulhich at gmail.com (MM) Date: Tue Sep 5 12:42:00 2023 Subject: [odb-users] after sqlite, choosing between postgresql and mysql Message-ID: On Sun, 3 Sept 2023 at 19:14, MM wrote: > Hello, > We are expanding our state in that we need to have a client server database > system to allow for concurrent read and write from different nodes. > > odb is finally paying off:-) > > This is a hard question but let me describe our use case > > 1. small number of concurrent users (under dozen) > 2. 3 databases in use, some read only, some read and write > 3. 2 dozen tables in each database > 4. Nbr of rows in each table ranges from 10 to 100 000 max > 5. Nbr of columns from 3/4 to 2 dozen columns, but only few tables > 6. concurrency: most of the transactions are read. 10% perhaps or less are > writes. > 7. There may be a desire to do a cross database "transaction", i.e. either > both databases are written successfully to or the transaction reverts on > both. > > That is what I can think of at the moment. Any advice? > > MM I realize it is not a closed ended question but if anyone has some guidelines as to how to pick from odb mysql vs odb postresql, either from experience or just generally, that would be appreciated rds, From anuragg at miltenyi.com Wed Sep 6 20:30:48 2023 From: anuragg at miltenyi.com (Anurag Gupta) Date: Thu Sep 7 06:29:27 2023 Subject: [odb-users] time_point to timestamp - pgsql? Message-ID: Hello, I read somewhere that odb natively supports std::chrono::system_clock::time_point but can't find the reference anywhere. I am trying to save date time using time_point as following: using datetime = std::chrono::system_clock::time_point; #pragma db value (datetime) type ("timestamp") // Is this correct? #pragma db object someClass { ... ... private: ??????datetime last_modified; ??????datetime created_date_time; ... } And after generating odb support code (using odb CLI) - here is the code snippet from someClass-odb.cxx: // last_modified // { ::datetime const& v = o.last_modified; bool is_null (false); pgsql::value_traits< ::datetime, pgsql::id_timestamp >::set_image ( //***** HERE I GET THE ERROR i.last_modified_value, is_null, v); i.last_modified_null = is_null; } // created_date_and_time // { ::datetime const& v = o.created_date_and_time; bool is_null (false); pgsql::value_traits< ::datetime, pgsql::id_timestamp >::set_image ( i.created_date_and_time_value, is_null, v); i.created_date_and_time_null = is_null; } Right when building it shows error: /usr/local/include/odb/pgsql/traits.hxx: In instantiation of ?static void odb::pgsql::default_value_traits >::set_image(image_type&, bool&, T) [with T = std::chrono::time_point > >; odb::pgsql::database_type_id ID = odb::pgsql::id_timestamp; image_type = long long int]?: ....../build/odb_gen/mbom_classes-odb.cxx:414:44: required from here??//****** THIS LINE POINTS TO ABOVE SOURCE LINE /usr/local/include/odb/pgsql/traits.hxx:373:43: error: invalid cast from type ?std::chrono::time_point > >? to type ?odb::pgsql::default_value_traits > >, odb::pgsql::id_timestamp>::image_type? {aka ?long long int?} 373 | i = details::endian_traits::hton (image_type (v)); | ^~~~~~~~~~~~~~ What am I doing wrong? How can I use std::chrono::system_clock::time_point in my code that can be persisted to PGSQL? Thank you! Anurag Gupta From boris at codesynthesis.com Thu Sep 7 07:04:17 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Sep 7 06:55:14 2023 Subject: [odb-users] time_point to timestamp - pgsql? In-Reply-To: References: Message-ID: Anurag Gupta writes: > using datetime = std::chrono::system_clock::time_point; > #pragma db value (datetime) type ("timestamp") // Is this correct? This will work but you will need to provide the value_traits implementation for converting time_point to the PosgreSQL TIMESTAMP type. Alternatively, you can try something easier like this: #pragma db map type(std::chrono::system_clock::time_point) as(std::uint64_t) \ to(std::chrono::duration_cast ( \ (?).time_since_epoch ()).count ()) \ from(std::chrono::system_clock::time_point ( \ std::chrono::duration_cast ( \ std::chrono::nanoseconds (?)))) This will store the time_point value as a number of nanoseconds since the UNIX epoch. Previously: https://codesynthesis.com/pipermail/odb-users/2019-September/004342.html From ryan at helloryan.se Thu Sep 7 06:55:31 2023 From: ryan at helloryan.se (Ryan Edin) Date: Thu Sep 7 06:56:14 2023 Subject: [odb-users] time_point to timestamp - pgsql? In-Reply-To: References: Message-ID: Hi! I would use the following type-mapping to store time_point as a uint64. // Map a time_point (datetime) to uint64_t. PRAGMA_DB( map type(datetime) as(std::uint64_t) to(::std::chrono::duration_cast<::std::chrono::milliseconds>((?).time_since_epoch()).count()) from(datetime(std::chrono::duration_cast(::std::chrono::milliseconds(?)))) ) // Ryan On Thu, Sep 7, 2023 at 12:38?PM Anurag Gupta wrote: > > Hello, > I read somewhere that odb natively supports std::chrono::system_clock::time_point but can't find the reference anywhere. I am trying to save date time using time_point as following: > > using datetime = std::chrono::system_clock::time_point; > #pragma db value (datetime) type ("timestamp") // Is this correct? > > > > > #pragma db object > someClass { > ... > ... > > private: > > datetime last_modified; > > datetime created_date_time; > > ... > } > > And after generating odb support code (using odb CLI) - here is the code snippet from someClass-odb.cxx: > > // last_modified > > // > > { > > ::datetime const& v = > > o.last_modified; > > > bool is_null (false); > > pgsql::value_traits< > > ::datetime, > > pgsql::id_timestamp >::set_image ( //***** HERE I GET THE ERROR > > i.last_modified_value, is_null, v); > > i.last_modified_null = is_null; > > } > > > // created_date_and_time > > // > > { > > ::datetime const& v = > > o.created_date_and_time; > > > bool is_null (false); > > pgsql::value_traits< > > ::datetime, > > pgsql::id_timestamp >::set_image ( > > i.created_date_and_time_value, is_null, v); > > i.created_date_and_time_null = is_null; > > } > > Right when building it shows error: > > /usr/local/include/odb/pgsql/traits.hxx: In instantiation of ?static void odb::pgsql::default_value_traits >::set_image(image_type&, bool&, T) [with T = std::chrono::time_point > >; odb::pgsql::database_type_id ID = odb::pgsql::id_timestamp; image_type = long long int]?: > ....../build/odb_gen/mbom_classes-odb.cxx:414:44: required from here??//****** THIS LINE POINTS TO ABOVE SOURCE LINE > /usr/local/include/odb/pgsql/traits.hxx:373:43: error: invalid cast from type ?std::chrono::time_point > >? to type ?odb::pgsql::default_value_traits > >, odb::pgsql::id_timestamp>::image_type? {aka ?long long int?} > 373 | i = details::endian_traits::hton (image_type (v)); > | ^~~~~~~~~~~~~~ > > What am I doing wrong? How can I use std::chrono::system_clock::time_point in my code that can be persisted to PGSQL? > > Thank you! > Anurag Gupta > > > From boris at codesynthesis.com Thu Sep 7 07:27:48 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Sep 7 07:18:46 2023 Subject: [odb-users] after sqlite, choosing between postgresql and mysql In-Reply-To: References: Message-ID: MM writes: > This is a hard question but let me describe our use case > > 1. small number of concurrent users (under dozen) > 2. 3 databases in use, some read only, some read and write > 3. 2 dozen tables in each database > 4. Nbr of rows in each table ranges from 10 to 100 000 max > 5. Nbr of columns from 3/4 to 2 dozen columns, but only few tables > 6. concurrency: most of the transactions are read. 10% perhaps or less are > writes. > 7. There may be a desire to do a cross database "transaction", i.e. either > both databases are written successfully to or the transaction reverts on > both. > > That is what I can think of at the moment. Any advice? I think both databases won't have any issues with points 1-6 on any reasonably decent hardware. Point 7 might get trickier. I have no experience with that on MySQL but for PG we have used foreign tables to achieve something like this. More generally, you commonly see the following points factored in when comparing the two databases: 1. PG is commonly viewed as a "saner" implementation of SQL. For example, it supports DDL statements within transactions. Another point for PG is support for bulk operations (which are now also supported in ODB). 2. MySQL, at least until recently, had better replication story, but I haven't tried replication in either so I cannot give any first- hand reports. 3. MySQL seems to have better concurrency story in that it is a multi-threaded database compared to multi-process for PG (but there are also reported advantages to multi-process, like better reliability). Personally, my choice would be (and has been) PostgreSQL because it aligns better with my view of what a database should feel like. Also, in my experience with supporting ODB, more serious use-cases tend to gravitate towards PG. If you want to see where we ourselves use the PG+ODB combo, that would be brep: https://git.build2.org/cgit/brep/ From d.patrushev at prosoftsystems.ru Thu Sep 21 09:19:55 2023 From: d.patrushev at prosoftsystems.ru (=?koi8-r?B?8MHU0tXbxdcg5MHOycwg4c7E0sXF18ne?=) Date: Thu Sep 21 09:11:00 2023 Subject: [odb-users] Prepared statement factory question Message-ID: <4e1cb830bec944f7a4eaf311a54f5f2c@prosoftsystems.ru> Hi. It looks like methods bool database::call_query_factory (const char* name, connection_type& c) const / void database::query_factory (const char* name, query_factory_wrapper w) are not thread-safe. Is it by design or an oversight? From boris at codesynthesis.com Fri Sep 22 03:08:43 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Sep 22 02:59:39 2023 Subject: [odb-users] Prepared statement factory question In-Reply-To: <4e1cb830bec944f7a4eaf311a54f5f2c@prosoftsystems.ru> References: <4e1cb830bec944f7a4eaf311a54f5f2c@prosoftsystems.ru> Message-ID: ???????? ????? ????????? writes: > It looks like methods > > bool database::call_query_factory (const char* name, connection_type& c) const > void database::query_factory (const char* name, query_factory_wrapper w) > > are not thread-safe. Is it by design or an oversight? Yes, this is by design. I've added[1] the following paragraph to the manual to make this explicit: "Note that the database::query_factory() function is not thread-safe and should be called before starting any threads that may require this functionality. Usually, all the prepared query factories are registered as part of the database instance creation." [1] https://git.codesynthesis.com/cgit/odb/odb/commit/?id=df4bbe5582d7