From reillyhekazusa at gmail.com Thu Feb 16 21:32:44 2023 From: reillyhekazusa at gmail.com (Reilly He) Date: Thu Feb 16 21:25:46 2023 Subject: [odb-users] Index Pragmas DO NOT support index on expressions Message-ID: Dear All, Recently I`m focusing on query optimization on our own project. 1. Env The general environment is: - *Compiler*: CXX11 - *OS*: Android&IOS - *Inner DB*: SQLite 2. Background information Here is basically the query looks like this: > *SELECT* * *FROM* `Post` > *WHERE* `group` = 1178615814 AND `isDirty` = 0 AND `isDeactivated` = 0 > *ORDER BY * > `id` < 0 *DESC*, > `creationTime` *DESC* > *LIMIT* 20 > *OFFSET* 0 > The query intends to look for at most 20 posts in a certain group/tribe in our db, and especially those posts with negative ids on the *first*, then the positive ids on the *latter*, and for these 2 groups, we sort it on `creationTime` respectively. In order to fully optimize this query, we need to bypass the additional Temp-BTree sorting on `creationTime` as well as the `id < 0` part. Since we are sorting based on the return of `id<0` (which is either 0 or 1), creating an index on data member `id` will not help. We think about several solutions on this: - Creating a new data member to interpret the negative or positive id situation: but that relies on a heavy data migration - Using 2 queries on (id < 0, id > 0), and then try to merge it: that compromises our LIMIT and OFFSET result - Using something like CASE/WHEN to group the data: that will be even harder to optimize - Using Index on expression: since SQLite supports it Obviously, *the index on expression solution is the best one*. 3. Problems But I found it really hard to create this index in the context of ODB index pragmas. I looked through the manual: https://www.codesynthesis.com/products/odb/doc/manual.xhtml#14.7 And I did not find any solution. *So, can you guys tell me how to create an index based on member expression in ODB? * *or does ODB support that?* So if ODB does not support creating indexes on an expression, I can do that on my own using SQLite CREATE INDEX pragma. But, this is not included in the ODB migration, which leaves a very unstable point in our project. Since if somebody in the future changes the schema in the Post table, this part might fail. *So, can you guys tell me how to adopt this manual changes into the ODB migration? * I have been stuck in this for several days. I would appreciate it a lot if you guys give me some suggestions. Thanks a lot. Best regards, Reilly He From boris at codesynthesis.com Fri Feb 17 01:40:36 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Feb 17 01:33:19 2023 Subject: [odb-users] Index Pragmas DO NOT support index on expressions In-Reply-To: References: Message-ID: Reilly He writes: > I found it really hard to create this index in the context of ODB index > pragmas. I looked through the manual: > https://www.codesynthesis.com/products/odb/doc/manual.xhtml#14.7 > And I did not find any solution. No, I don't think there is way to achieve this via the pragma currently. > So if ODB does not support creating indexes on an expression, I can do that > on my own using SQLite CREATE INDEX pragma. Yes, this is probably the only way, currently. > But, this is not included in the ODB migration, which leaves a very > unstable point in our project. Since if somebody in the future changes the > schema in the Post table, this part might fail. Yes, that's a valid concern. However, even if we were to add support for this in the pragma, it will be just as a raw column name (or column expression). Something along these lines: class Post { int id; #pragma db index("id_i") column("abs(id)") }; So there is still a possibility of, say, renaming the id member but forgetting to update the index definition. Though having the two next to each other definitely reduces this likelihood compared to creating the index manually. If you would like, we can add this support to the index pragma. From reillyhekazusa at gmail.com Fri Feb 17 04:46:44 2023 From: reillyhekazusa at gmail.com (Reilly He) Date: Fri Feb 17 04:39:46 2023 Subject: [odb-users] Index Pragmas DO NOT support index on expressions In-Reply-To: References: Message-ID: Hi Boris, Thanks a lot for your quick response. So there is still a possibility of, say, renaming the id member but > forgetting to update the index definition. > I got to say I am a little bit confused about this statement, since I take a look at the docs of SQLite Alter Table , and here I quote: The RENAME COLUMN TO syntax changes the column-name of table table-name > into new-column-name. The column name is changed both within the table > definition itself *and also within all indexes*, triggers, and views that > reference the column. If the column name change would result in a semantic > ambiguity in a trigger or view, then the RENAME COLUMN fails with an error > and no changes are applied. > Actually I`ve done my experiment today, and the saying of SQLite was *TRUE*, so things like changing the column name actually will not fail any index including the changed column. but according to your statement, are you saying that if we already add an index using ODB`s index pragma, and we somehow *changed the member name* (column name), the compiling process of ODB will *fail*? If you would like, we can add this support to the index pragma. > I`m not in a rush, since we decided to implement this index using the SQLite way, but it is always good to have it LOL, since some optimization will be pretty tricky if we do not have this. Best Regards, Reilly He On Fri, Feb 17, 2023 at 2:40 PM Boris Kolpackov wrote: > Reilly He writes: > > > I found it really hard to create this index in the context of ODB index > > pragmas. I looked through the manual: > > https://www.codesynthesis.com/products/odb/doc/manual.xhtml#14.7 > > And I did not find any solution. > > No, I don't think there is way to achieve this via the pragma currently. > > > > So if ODB does not support creating indexes on an expression, I can do > that > > on my own using SQLite CREATE INDEX pragma. > > Yes, this is probably the only way, currently. > > > > But, this is not included in the ODB migration, which leaves a very > > unstable point in our project. Since if somebody in the future changes > the > > schema in the Post table, this part might fail. > > Yes, that's a valid concern. However, even if we were to add support > for this in the pragma, it will be just as a raw column name (or column > expression). Something along these lines: > > class Post > { > int id; > > #pragma db index("id_i") column("abs(id)") > }; > > So there is still a possibility of, say, renaming the id member but > forgetting to update the index definition. Though having the two > next to each other definitely reduces this likelihood compared to > creating the index manually. > > If you would like, we can add this support to the index pragma. > From boris at codesynthesis.com Fri Feb 17 05:19:59 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Feb 17 05:12:43 2023 Subject: [odb-users] Index Pragmas DO NOT support index on expressions In-Reply-To: References: Message-ID: Reilly He writes: > > So there is still a possibility of, say, renaming the id member but > > forgetting to update the index definition. > > > > I got to say I am a little bit confused about this statement, since I take > a look at the docs of SQLite Alter Table > , and here I quote: > > > The RENAME COLUMN TO syntax changes the column-name of table table-name > > into new-column-name. The column name is changed both within the table > > definition itself *and also within all indexes*, triggers, and views that > > reference the column. If the column name change would result in a semantic > > ambiguity in a trigger or view, then the RENAME COLUMN fails with an error > > and no changes are applied. > > Actually I`ve done my experiment today, and the saying of SQLite was *TRUE*, > so things like changing the column name actually will not fail any index > including the changed column. > > but according to your statement, are you saying that if we already add an > index using ODB`s index pragma, and we somehow *changed the member name* > (column name), the compiling process of ODB will *fail*? No, it's in a sense the opposite: if you use raw column names and there is a mismatch, then the ODB compiler is not going to fail and you will likely get a runtime error later. In the example I've given SQLite is smart enough to take care of things if you are changing the schema. But you could also be creating the schema (with mismatched column/index) from scratch, say, in a new database. In this case I am pretty certain you will get a runtime error since the index will refer to the column that does not exist. From reillyhekazusa at gmail.com Fri Feb 17 07:22:37 2023 From: reillyhekazusa at gmail.com (Reilly He) Date: Fri Feb 17 07:15:37 2023 Subject: [odb-users] Index Pragmas DO NOT support index on expressions In-Reply-To: References: Message-ID: Ok, thanks for the clarification. On Fri, Feb 17, 2023 at 6:20 PM Boris Kolpackov wrote: > Reilly He writes: > > > > So there is still a possibility of, say, renaming the id member but > > > forgetting to update the index definition. > > > > > > > I got to say I am a little bit confused about this statement, since I > take > > a look at the docs of SQLite Alter Table > > , and here I quote: > > > > > The RENAME COLUMN TO syntax changes the column-name of table table-name > > > into new-column-name. The column name is changed both within the table > > > definition itself *and also within all indexes*, triggers, and views > that > > > reference the column. If the column name change would result in a > semantic > > > ambiguity in a trigger or view, then the RENAME COLUMN fails with an > error > > > and no changes are applied. > > > > Actually I`ve done my experiment today, and the saying of SQLite was > *TRUE*, > > so things like changing the column name actually will not fail any index > > including the changed column. > > > > but according to your statement, are you saying that if we already add an > > index using ODB`s index pragma, and we somehow *changed the member name* > > (column name), the compiling process of ODB will *fail*? > > No, it's in a sense the opposite: if you use raw column names and there > is a mismatch, then the ODB compiler is not going to fail and you will > likely get a runtime error later. > > In the example I've given SQLite is smart enough to take care of things > if you are changing the schema. But you could also be creating the > schema (with mismatched column/index) from scratch, say, in a new > database. In this case I am pretty certain you will get a runtime > error since the index will refer to the column that does not exist. > From antoine.gennart at quimesis.be Thu Feb 23 02:48:29 2023 From: antoine.gennart at quimesis.be (Antoine Gennart) Date: Thu Feb 23 03:35:04 2023 Subject: [odb-users] Compile libodb for c++17 Message-ID: Hello, I am currently facing a problem while compiling the library libodb using c++17. Libobd is currently using a removed feature of c++ which is the dynamic exception specification. Below, I copied a small example code to explain the problem of the current implementation. Therefore I was wondering if there a plan to upgrade your library to newer standard ? As far as I understand this error, this would be as difficult as to remove the "throw"statement next to the function definition, and it would also be nice to add a "noexcept" qualifier for the functions that cannot throw errors. Thank you for your time, Best regards, Antoine Gennart g++ -std=c++17 main.cpp -o main ``` #include ? // does compile ? void test_throw_1() ? { ????? throw(1); ? } ? // does compile ? // will call std::terminate when exception is throwed ? void test_throw_2() noexcept ? { --??? throw(1); ? } ? // does not compile ? // deprecated since c++11 ? // removed since c++17 --void test_throw_3() throw(int) ? { ????? throw(1); ? } ? int main (int argc, char *argv[]) ? { ????? std::cout << "Hello world" << std::endl; ????? try ????? { ????????? test_throw_1(); ????????? test_throw_2(); ????????? test_throw_3(); ????? } ????? catch (int& i) ????? { ????????? std::cout << i << std::endl; ????? } ????? return 0; ? } ``` From boris at codesynthesis.com Thu Feb 23 04:57:58 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Feb 23 04:50:36 2023 Subject: [odb-users] Compile libodb for c++17 In-Reply-To: References: Message-ID: Antoine Gennart writes: > I am currently facing a problem while compiling the library libodb using > c++17. Libobd is currently using a removed feature of c++ which is the > dynamic exception specification. > > Therefore I was wondering if there a plan to upgrade your library to newer > standard ? Yes, this has already been done for the upcoming release (2.5.0). If you would like, you can start using the pre-release as described on this page: https://codesynthesis.com/products/odb/doc/install-build2.xhtml From iherna31 at asu.edu Thu Feb 23 22:20:26 2023 From: iherna31 at asu.edu (Irvin Josue Hernandez Rivera) Date: Thu Feb 23 22:31:54 2023 Subject: [odb-users] Creating virtual tables Message-ID: Hello, I am currently trying to use spatial indexes for some Geospatial lookups that I need to implement for a DB in c++. Does odb allow the creation of virtual tables with a praga definition? Regards, Josue Hernandez From boris at codesynthesis.com Mon Feb 27 07:33:43 2023 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Feb 27 07:26:20 2023 Subject: [odb-users] Creating virtual tables In-Reply-To: References: Message-ID: Irvin Josue Hernandez Rivera writes: > I am currently trying to use spatial indexes for some Geospatial lookups > that I need to implement for a DB in c++. Does odb allow the creation of > virtual tables with a praga definition? It's hard to say anything specific since you didn't mention the underlying database you are using. However, we've recently added support for the table `options` pragma that allows customizing the table definitions generated by ODB. Perhaps it will allow you to achieve what you want: https://git.codesynthesis.com/cgit/odb/odb/commit/?id=5652354aa256426c5ab32c4a1c5687e5509af868 From iherna31 at asu.edu Mon Feb 27 10:30:32 2023 From: iherna31 at asu.edu (Irvin Josue Hernandez Rivera) Date: Tue Feb 28 08:24:58 2023 Subject: [odb-users] Creating virtual tables In-Reply-To: References: Message-ID: Thank you for your response. The underlying DB I am using is Sqlite DB and I am using this db to store user positions as a Lat Lon. I wanted to add support to the DB for ranged queries an rtree https://www.sqlite.org/rtree.html In order for sqlite to create the rtree module the table instantiation need to be as follows: CREATE VIRTUAL TABLE ** USING rtree(**); I don't think the options pragma allows for this type of table creation On Mon, Feb 27, 2023, 5:33 AM Boris Kolpackov wrote: > Irvin Josue Hernandez Rivera writes: > > > I am currently trying to use spatial indexes for some Geospatial lookups > > that I need to implement for a DB in c++. Does odb allow the creation of > > virtual tables with a praga definition? > > It's hard to say anything specific since you didn't mention the underlying > database you are using. However, we've recently added support for the > table `options` pragma that allows customizing the table definitions > generated by ODB. Perhaps it will allow you to achieve what you want: > > > https://urldefense.com/v3/__https://git.codesynthesis.com/cgit/odb/odb/commit/?id=5652354aa256426c5ab32c4a1c5687e5509af868__;!!IKRxdwAv5BmarQ!e6FfD6qB-KsqOvHZ-ijvz3m7Gd1MVvuwYXv-pB02G_ErL1KbqVoaq3B0eGY0OKK6kL-0DdMJB3BDvm74wd4$ >