From cezanmeyen at gmail.com Thu Apr 2 08:52:42 2020 From: cezanmeyen at gmail.com (Cezan Meyen) Date: Thu Apr 2 09:01:55 2020 Subject: [odb-users] Question regarding odb::timeout Message-ID: Hello, I?m using ODB with SQLite(in tmpfs) for multi-process data storage and exchange. It is working fine but I?ve encountered a lot of odb::timeout exceptions when multiple processes are writing at the same time. My initial solution was simply retrying every time it threw the exception(as the ODB doc states). This resulted in a lot of retries per persist and a significant performance impact. My second and current solution is using an inter-process mutex on the persist actions. This will ensure that there is only a single process is writing data at a given time, so no timeout exceptions are thrown. This performed better when compared to the first solution and seemed a good solution at the time. It now however poses a big problem when scaling up with more and more concurrent writers. Since the file is locked the wait times increase with the amount of persist processes. I?ve run tests with MySQL and PostgreSQL as they are made to handle multi-process data exchange better. But they both performed worse than my SQLite + mutex solution. In these tests they were both mounted in tmpfs since their in-memory tables weren?t usable for out use-case. My questions are: - When exactly does the odb::timeout exception get thrown? - Is there a way to setup odb and sqlite to handle multi-process writers more consistent? o I am already working with WAL - Does ODB use the locks provided by the sqlite3 API? o If not is it possible to interact with these locks through the ODB API or should this be done natively? A more general question: - Is there a different way to deal with this congestion problem, that you know of? Kind regards, C?zan From boris at codesynthesis.com Fri Apr 3 06:29:00 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Apr 3 06:37:59 2020 Subject: [odb-users] Question regarding odb::timeout In-Reply-To: References: Message-ID: Cezan Meyen writes: > I?m using ODB with SQLite(in tmpfs) for multi-process data storage and > exchange. It is working fine but I?ve encountered a lot of odb::timeout > exceptions when multiple processes are writing at the same time. My initial > solution was simply retrying every time it threw the exception(as the ODB > doc states). This resulted in a lot of retries per persist and a > significant performance impact. In SQLite a write transaction needs exclusive access to the database. If it is already used by another transaction SQLite returns SQLITE_BUSY which ODB translates to the odb::timeout exception. See the SQLITE_BUSY[1] documentation for details. > My second and current solution is using an inter-process mutex on the > persist actions. This will ensure that there is only a single process is > writing data at a given time, so no timeout exceptions are thrown. This > performed better when compared to the first solution and seemed a good > solution at the time. It now however poses a big problem when scaling up > with more and more concurrent writers. Since the file is locked the wait > times increase with the amount of persist processes. Yes, SQLite does not handle write-heavy, high-contention workloads well. You could probably improve the performance for the multiple readers case by using the reader-writer mutex but it will always be a single writer per the entire database at a time. To elaborate on the write-heavy, high-contention workloads, we have a test in the odb-tests/common/thread[2] that can be used to benchmark this type of load. It also contains some SQLite-specific optimizations which you may find helpful (like starting the write transaction as such). There is also an admittedly old blog post[3] that besides other things shows the results of this benchmark (all the way at the end). While it was tested on older hardware, I believe relative values and general observations still hold. > I?ve run tests with MySQL and PostgreSQL as they are made to handle > multi-process data exchange better. But they both performed worse than > my SQLite + mutex solution. That would only be the case up to a certain load/core count. As the above-mentioned blog post demonstrates, they will outperform SQLite as things get more contentious. > - Is there a way to setup odb and sqlite to handle multi-process > writers more consistent? > > - Does ODB use the locks provided by the sqlite3 API? Yes, ODB uses the SQLite shared cache mode and the unlock notify functionality to aid concurrency but that only works within a single multi-threaded process. While I would expect it to have less overhead compared to the file/inter-process locking, you still have the same restriction of only one write transaction at a time. > - Is there a different way to deal with this congestion problem, > that you know of? I don't believe much can be done about SQLite's poor write-heavy, high-contention performance (other than using a different database). Which only leaves optimizations on your application's side. I think the following will be the most promising optimization directions: 1. Reduce the number of database statements executed (e.g, mapping an array as a BLOB). 2. Reduce the amount of data manipulated by each statement (using sections, lazy pointers, views, etc). 3. Reduce contention so more statements can be executed in parallel (rw mutex, making transactions as short and fast as possible without doing any unnecessary work there, etc). To elaborate, storing arrays, especially arrays like 1000s of doubles, in a single column rather than in a separate table is definitely a good idea. The SQLite type system is very limited and there are no array types (unlike, for example, PostgreSQL). However, I don't believe this would be any more efficient than a BLOB. Another powerful optimization technique worth considering are sections (Chapter 9). For example, if the above array of doubles (stored as a BLOB) is not always needed, it can be placed into a separate "section" of the object and loaded only if and when necessary. In a similar vein, for object relationship there is are lazy pointers (Section 6.4) which avoid loading the pointed-to object until necessary. For query optimization, there are views (Chapter 10) which allow you to only load a subset of data members, load from multiple objects with a single statement, etc. Finally, if you do apply these optimizations, I would recommend to also re-test client-server databases (PostgreSQL, MySQL) since these kinds of improvements (especially reducing the number of statements executed) will benefit them a lot more than SQLite due to their client-server nature. [1] https://www.sqlite.org/rescode.html#busy [2] https://git.codesynthesis.com/cgit/odb/odb-tests/tree/common/threads [3] https://www.codesynthesis.com/~boris/blog/2012/02/02/updated-odb-benchmark-results/ From oliverjfrancis at gmail.com Sat Apr 4 00:44:16 2020 From: oliverjfrancis at gmail.com (Oliver Francis) Date: Mon Apr 6 08:18:25 2020 Subject: [odb-users] Error invoking odb 2.4.0 with g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Message-ID: <7637E905-C060-4676-8630-62B6D49213DD@gmail.com> Hi, there. I?ve tried compiling a simple odb file with odb 2.4.0 and g++ 7.5.0, but I get the following error: odb -d mysql -o asp/config --generate-query --generate-schema --hxx-suffix .h --ixx-suffix .inl --cxx-suffix .cpp --odb-file-suffix _odb --std c++14 -I./ asp/config/gtt_config.odb cc1plus: error: bad value (?/dev/null?) for ?-mtune=? switch cc1plus: note: valid arguments to ?-mtune=? switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 bonnell atom silvermont slm knl intel x86-64 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 btver1 btver2 generic The verbose output can be found below: odb -v -d mysql -o asp/config --generate-query --generate-schema --hxx-suffix .h --ixx-suffix .inl --cxx-suffix .cpp --odb-file-suffix _odb --std c++14 -I./ asp/config/gtt_config.odb Profile search paths: . /usr/include/c++/7 /usr/include/x86_64-linux-gnu/c++/7 /usr/include/c++/7/backward /usr/lib/gcc/x86_64-linux-gnu/7/include /usr/local/include /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/include/x86_64-linux-gnu /usr/include Compiling asp/config/gtt_config.odb g++ -x c++ -std=gnu++14 -S -Wunknown-pragmas -Wno-deprecated -fplugin=/usr/lib/gcc/x86_64-linux-gnu/7/plugin/odb.so -v -I./ -fplugin-arg-odb-svc-path=. -fplugin-arg-odb-svc-path=/usr/include/c++/7 -fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu/c++/7 -fplugin-arg-odb-svc-path=/usr/include/c++/7/backward -fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include -fplugin-arg-odb-svc-path=/usr/local/include -fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu -fplugin-arg-odb-svc-path=/usr/include -DODB_COMPILER -DODB_COMPILER_VERSION=2040000 -DODB_DATABASE_MYSQL -fplugin-arg-odb-database=mysql -fplugin-arg-odb-o=asp/config -fplugin-arg-odb-generate-query -fplugin-arg-odb-generate-schema -fplugin-arg-odb-hxx-suffix=.h -fplugin-arg-odb-ixx-suffix=.inl -fplugin-arg-odb-cxx-suffix=.cpp -fplugin-arg-odb-odb-file-suffix=_odb -fplugin-arg-odb-std=c++14 -fplugin-arg-odb-svc-file=asp/config/gtt_config.odb - Using built-in specs. COLLECT_GCC=g++ OFFLOAD_TARGET_NAMES=nvptx-none OFFLOAD_TARGET_DEFAULT=1 Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.5.0-3ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04) COLLECT_GCC_OPTIONS='-std=gnu++14' '-S' '-Wunknown-pragmas' '-Wno-deprecated' '-fplugin=/usr/lib/gcc/x86_64-linux-gnu/7/plugin/odb.so' '-v' '-I' './' '-fplugin-arg-odb-svc-path=.' '-fplugin-arg-odb-svc-path=/usr/include/c++/7' '-fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu/c++/7' '-fplugin-arg-odb-svc-path=/usr/include/c++/7/backward' '-fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include' '-fplugin-arg-odb-svc-path=/usr/local/include' '-fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed' '-fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu' '-fplugin-arg-odb-svc-path=/usr/include' '-D' 'ODB_COMPILER' '-D' 'ODB_COMPILER_VERSION=2040000' '-D' 'ODB_DATABASE_MYSQL' '-fplugin-arg-odb-database=mysql' '-fplugin-arg-odb-o=asp/config' '-fplugin-arg-odb-generate-query' '-fplugin-arg-odb-generate-schema' '-fplugin-arg-odb-hxx-suffix=.h' '-fplugin-arg-odb-ixx-suffix=.inl' '-fplugin-arg-odb-cxx-suffix=.cpp' '-fplugin-arg-odb-odb-file-suffix=_odb' '-fplugin-arg-odb-std=c++14' '-fplugin-arg-odb-svc-file=asp/config/gtt_config.odb' '-shared-libgcc' '-mtune=generic' '-march=x86-64' /usr/lib/gcc/x86_64-linux-gnu/7/cc1plus -quiet -v -I ./ -imultiarch x86_64-linux-gnu -iplugindir=/usr/lib/gcc/x86_64-linux-gnu/7/plugin -D_GNU_SOURCE -D ODB_COMPILER -D ODB_COMPILER_VERSION=2040000 -D ODB_DATABASE_MYSQL - -iplugindir=/usr/lib/gcc/x86_64-linux-gnu/7/plugin -quiet -dumpbase - -mtune=generic -march=x86-64 -auxbase - -Wunknown-pragmas -Wno-deprecated -std=gnu++14 -version -fplugin=/usr/lib/gcc/x86_64-linux-gnu/7/plugin/odb.so -fplugin-arg-odb-svc-path=. -fplugin-arg-odb-svc-path=/usr/include/c++/7 -fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu/c++/7 -fplugin-arg-odb-svc-path=/usr/include/c++/7/backward -fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include -fplugin-arg-odb-svc-path=/usr/local/include -fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu -fplugin-arg-odb-svc-path=/usr/include -fplugin-arg-odb-database=mysql -fplugin-arg-odb-o=asp/config -fplugin-arg-odb-generate-query -fplugin-arg-odb-generate-schema -fplugin-arg-odb-hxx-suffix=.h -fplugin-arg-odb-ixx-suffix=.inl -fplugin-arg-odb-cxx-suffix=.cpp -fplugin-arg-odb-odb-file-suffix=_odb -fplugin-arg-odb-std=c++14 -fplugin-arg-odb-svc-file=asp/config/gtt_config.odb -o -.s -fstack-protector-strong -Wformat -Wformat-security GNU C++14 (Ubuntu 7.5.0-3ubuntu1~18.04) version 7.5.0 (x86_64-linux-gnu) compiled by GNU C version 7.5.0, GMP version 6.1.2, MPFR version 4.0.1, MPC version 1.1.0, isl version isl-0.19-GMP GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072 Versions of loaded plugins: odb: 2.4.0 ignoring duplicate directory "/usr/include/x86_64-linux-gnu/c++/7" ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu" ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/7/../../../../x86_64-linux-gnu/include" #include "..." search starts here: #include <...> search starts here: ./ /usr/include/c++/7 /usr/include/x86_64-linux-gnu/c++/7 /usr/include/c++/7/backward /usr/lib/gcc/x86_64-linux-gnu/7/include /usr/local/include /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/include/x86_64-linux-gnu /usr/include End of search list. cc1plus: error: bad value (?/dev/null?) for ?-mtune=? switch cc1plus: note: valid arguments to ?-mtune=? switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 bonnell atom silvermont slm knl intel x86-64 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 btver1 btver2 generic GNU C++14 (Ubuntu 7.5.0-3ubuntu1~18.04) version 7.5.0 (x86_64-linux-gnu) compiled by GNU C version 7.5.0, GMP version 6.1.2, MPFR version 4.0.1, MPC version 1.1.0, isl version isl-0.19-GMP GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072 Versions of loaded plugins: odb: 2.4.0 This command used to work fine on another .odb file a few months ago when I last used odb successfully. I also attempted to compile my previous .odb files and that also failed. I suspect that odb 2.4.0 is incompatible with g++ 7.5.0, as I?m pretty sure that I used to have 7.4.0 installed and that work. Please would you mind letting me know what my best course of action would be? Best regards, Oliver From boris at codesynthesis.com Mon Apr 6 08:13:14 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Apr 6 08:22:22 2020 Subject: [odb-users] Error invoking odb 2.4.0 with g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 In-Reply-To: <7637E905-C060-4676-8630-62B6D49213DD@gmail.com> References: <7637E905-C060-4676-8630-62B6D49213DD@gmail.com> Message-ID: Oliver Francis writes: > cc1plus: error: bad value (?/dev/null?) for ?-mtune=? switch https://www.codesynthesis.com/pipermail/odb-users/2020-March/004469.html From silversbn at gmail.com Tue Apr 7 02:03:52 2020 From: silversbn at gmail.com (Sergey Bazhnev) Date: Tue Apr 7 05:05:21 2020 Subject: [odb-users] ODB compiler error Message-ID: Hello! Thank you for your useful system! But, recently by compiling I get the following message: cc1plus: error: bad value (?/dev/null?) for ?-mtune=? switch cc1plus: note: valid arguments to ?-mtune=? switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 bonnell atom silvermont slm knl intel x86-64 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 btver1 btver2 generic My system: ODB object-relational mapping (ORM) compiler for C++ 2.4.0 g++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Verbose output: g++ -x c++ -std=gnu++0x -S -Wunknown-pragmas -Wno-deprecated -fplugin=/usr/lib/gcc/x86_64-linux-gnu/7/plugin/odb.so -v -fplugin-arg-odb-svc-path=/usr/include/c++/7 -fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu/c++/7 -fplugin-arg-odb-svc-path=/usr/include/c++/7/backward -fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include -fplugin-arg-odb-svc-path=/usr/local/include -fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu -fplugin-arg-odb-svc-path=/usr/include -DODB_COMPILER -DODB_COMPILER_VERSION=2040000 -DODB_DATABASE_SQLITE -fplugin-arg-odb-database=sqlite -fplugin-arg-odb-std=c++11 -fplugin-arg-odb-q -fplugin-arg-odb-s -fplugin-arg-odb-svc-file=./Transfer.h - Using built-in specs. COLLECT_GCC=g++ OFFLOAD_TARGET_NAMES=nvptx-none OFFLOAD_TARGET_DEFAULT=1 Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.5.0-3ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04) COLLECT_GCC_OPTIONS='-std=gnu++11' '-S' '-Wunknown-pragmas' '-Wno-deprecated' '-fplugin=/usr/lib/gcc/x86_64-linux-gnu/7/plugin/odb.so' '-v' '-fplugin-arg-odb-svc-path=/usr/include/c++/7' '-fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu/c++/7' '-fplugin-arg-odb-svc-path=/usr/include/c++/7/backward' '-fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include' '-fplugin-arg-odb-svc-path=/usr/local/include' '-fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed' '-fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu' '-fplugin-arg-odb-svc-path=/usr/include' '-D' 'ODB_COMPILER' '-D' 'ODB_COMPILER_VERSION=2040000' '-D' 'ODB_DATABASE_SQLITE' '-fplugin-arg-odb-database=sqlite' '-fplugin-arg-odb-std=c++11' '-fplugin-arg-odb-q' '-fplugin-arg-odb-s' '-fplugin-arg-odb-svc-file=./Transfer.h' '-shared-libgcc' '-mtune=generic' '-march=x86-64' /usr/lib/gcc/x86_64-linux-gnu/7/cc1plus -quiet -v -imultiarch x86_64-linux-gnu -iplugindir=/usr/lib/gcc/x86_64-linux-gnu/7/plugin -D_GNU_SOURCE -D ODB_COMPILER -D ODB_COMPILER_VERSION=2040000 -D ODB_DATABASE_SQLITE - -iplugindir=/usr/lib/gcc/x86_64-linux-gnu/7/plugin -quiet -dumpbase - -mtune=generic -march=x86-64 -auxbase - -Wunknown-pragmas -Wno-deprecated -std=gnu++11 -version -fplugin=/usr/lib/gcc/x86_64-linux-gnu/7/plugin/odb.so -fplugin-arg-odb-svc-path=/usr/include/c++/7 -fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu/c++/7 -fplugin-arg-odb-svc-path=/usr/include/c++/7/backward -fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include -fplugin-arg-odb-svc-path=/usr/local/include -fplugin-arg-odb-svc-path=/usr/lib/gcc/x86_64-linux-gnu/7/include-fixed -fplugin-arg-odb-svc-path=/usr/include/x86_64-linux-gnu -fplugin-arg-odb-svc-path=/usr/include -fplugin-arg-odb-database=sqlite -fplugin-arg-odb-std=c++11 -fplugin-arg-odb-q -fplugin-arg-odb-s -fplugin-arg-odb-svc-file=./Transfer.h -o -.s -fstack-protector-strong -Wformat -Wformat-security GNU C++11 (Ubuntu 7.5.0-3ubuntu1~18.04) version 7.5.0 (x86_64-linux-gnu) compiled by GNU C version 7.5.0, GMP version 6.1.2, MPFR version 4.0.1, MPC version 1.1.0, isl version isl-0.19-GMP GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072 Versions of loaded plugins: odb: 2.4.0 ignoring duplicate directory "/usr/include/x86_64-linux-gnu/c++/7" ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu" ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/7/../../../../x86_64-linux-gnu/include" #include "..." search starts here: #include <...> search starts here: /usr/include/c++/7 /usr/include/x86_64-linux-gnu/c++/7 /usr/include/c++/7/backward /usr/lib/gcc/x86_64-linux-gnu/7/include /usr/local/include /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed /usr/include/x86_64-linux-gnu /usr/include End of search list. cc1plus: error: bad value (?/dev/null?) for ?-mtune=? switch cc1plus: note: valid arguments to ?-mtune=? switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 bonnell atom silvermont slm knl intel x86-64 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 btver1 btver2 generic GNU C++11 (Ubuntu 7.5.0-3ubuntu1~18.04) version 7.5.0 (x86_64-linux-gnu) compiled by GNU C version 7.5.0, GMP version 6.1.2, MPFR version 4.0.1, MPC version 1.1.0, isl version isl-0.19-GMP GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072 Versions of loaded plugins: odb: 2.4.0 I don't know how I can fix it. Please, help! Same problem https://stackoverflow.com/questions/60762771/what-causes-gcc-invalid-option-for-mtune Thanks! -- At your service, Sergey Bazhnev. From boris at codesynthesis.com Tue Apr 7 05:01:46 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Apr 7 05:10:55 2020 Subject: [odb-users] ODB compiler error In-Reply-To: References: Message-ID: Sergey Bazhnev writes: > Thank you for your useful system! Thanks, I am glad you find it useful. > But, recently by compiling I get the following message: > cc1plus: error: bad value (?/dev/null?) for ?-mtune=? switch Please see: https://www.codesynthesis.com/pipermail/odb-users/2020-March/004469.html Can you also please comment/upvote this Ubuntu bug (there is "This bug affects you" link under the title): https://bugs.launchpad.net/bugs/1871095 From haupt.wolfgang at gmail.com Mon Apr 13 11:27:26 2020 From: haupt.wolfgang at gmail.com (Wolfgang Haupt) Date: Mon Apr 13 11:37:06 2020 Subject: [odb-users] ODB on Android Message-ID: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> Hi, I'm trying to get odb working for android. So far I read: http://wiki.codesynthesis.com/Using_ODB_on_Android And I can cross-compile libodb and libodb-sqlite+libodb-mysql just fine. However I struggle to see how this will work together with the odb compiler. In the above guide there is no word about the odb compiler itself, so I tried to use the binary distribution for linux. When I do this I have to patch: /arm-linux-androideabi-21-debug/lib/odb/x86_64-linux-gnu/include/c++/4.9.3/x86_64-linux-gnu/bits/os_defines.h/ Because /__GLIBC_PREREQ/ is not defined. Then I get more errors: /In file included from /home/a1rwulf/Android/android-ndk-r18b/sysroot/usr/include/features.h:33:0,// //???????????????? from /home/a1rwulf/Android/omniyon-depends/arm-linux-androideabi-21-debug/lib/odb/x86_64-linux-gnu/include/c++/4.9.3/x86_64-linux-gnu/bits/os_defines.h:39,// //???????????????? from /home/a1rwulf/Android/omniyon-depends/arm-linux-androideabi-21-debug/lib/odb/x86_64-linux-gnu/include/c++/4.9.3/x86_64-linux-gnu/bits/c++config.h:430,// //???????????????? from /home/a1rwulf/Android/omniyon-depends/arm-linux-androideabi-21-debug/lib/odb/x86_64-linux-gnu/include/c++/4.9.3/string:38,// //???????????????? from :7:// ///home/a1rwulf/Android/android-ndk-r18b/sysroot/usr/include/string.h:205:9: error: attributes are not allowed on a function-definition// //???????? __prefer_this_overload {/ I'm pretty sure it is because the compiled gcc included in the binary distribution package does of course target Linux and not Android and there will be a mismatch in the libc or libstdc++ etc. (I lack knowledge in this area) I then tried to compile odb for Android, but I don't really see how this works as Android has switched to clang and the toolchain does not have a gcc with plugin support. Any ideas on how to proceed with this? Did anybody already get this working with recent Android NDK's (I'm using NDK 18b atm)? Best Regards, Wolfgang From boris at codesynthesis.com Tue Apr 14 06:02:45 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Apr 14 06:12:17 2020 Subject: [odb-users] ODB on Android In-Reply-To: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> References: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> Message-ID: Wolfgang Haupt writes: > And I can cross-compile libodb and libodb-sqlite+libodb-mysql just fine. > > However I struggle to see how this will work together with the odb compiler. You would normally use native ODB compiler for your development machine. So if you are using x86-64 Linux to develop for ARM Android, then you would run x86-64 Linux ODB compiler, generate the database support source code (the source code is the same for all the platforms), then compile that for ARM Android along with your other source code, and finally link everything together with cross-compiled libodb*. It may be possible to build a native ODB compiler for Android but we haven't tried (you will most likely need to build GCC from scratch with plugin support). From haupt.wolfgang at gmail.com Tue Apr 14 08:45:19 2020 From: haupt.wolfgang at gmail.com (Wolfgang Haupt) Date: Tue Apr 14 08:54:59 2020 Subject: [odb-users] ODB on Android In-Reply-To: References: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> Message-ID: <6f744849-b562-06e9-85c7-5654c4079568@gmail.com> On 14.04.20 12:02, Boris Kolpackov wrote: > Wolfgang Haupt writes: > >> And I can cross-compile libodb and libodb-sqlite+libodb-mysql just fine. >> >> However I struggle to see how this will work together with the odb compiler. > You would normally use native ODB compiler for your development machine. > So if you are using x86-64 Linux to develop for ARM Android, then you > would run x86-64 Linux ODB compiler, generate the database support > source code (the source code is the same for all the platforms), then > compile that for ARM Android along with your other source code, and > finally link everything together with cross-compiled libodb*. > > It may be possible to build a native ODB compiler for Android but we > haven't tried (you will most likely need to build GCC from scratch > with plugin support). Hey Boris, thx for your clarification - seems the procedure works better when I use version 17 of the android ndk. I'm still in the compilation process, so I can probably report back once it is finished. Given the earlier mentioned errors just disappear when using the 17 NDK, I wonder how/if it is possible for the odb compiler (on the host machine) to generate valid c++ code for the target, when the target probably uses a completely different standard c/c++ library and has no support for GCC at all. > It may be possible to build a native ODB compiler for Android but we > haven't tried (you will most likely need to build GCC from scratch > with plugin support). I do not want to go this way either if not possbile, sounds like pain to me. Does that mean you have tried to use odb on a recent android ndk >= 18? Best Regards, Wolfgang From c.sell at byterefinery.de Tue Apr 14 08:56:59 2020 From: c.sell at byterefinery.de (c.sell@byterefinery.de) Date: Tue Apr 14 09:06:38 2020 Subject: [odb-users] ODB on Android In-Reply-To: <6f744849-b562-06e9-85c7-5654c4079568@gmail.com> References: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> <6f744849-b562-06e9-85c7-5654c4079568@gmail.com> Message-ID: <20200414125659.Horde.VS7JP4VXzisswZbqZShUM-2@webmail.byterefinery.de> Hi, IMO and according to my experience, the ODB compiler has no dependency on the target system, including the c/c++ libraries. After all, it is just a code generator. As long as it generates valid standard C++ code, it should work on any system. I have compiled on ealier NDKs without issues. Chris Zitat von Wolfgang Haupt : > On 14.04.20 12:02, Boris Kolpackov wrote: >> Wolfgang Haupt writes: >> >>> And I can cross-compile libodb and libodb-sqlite+libodb-mysql just fine. >>> >>> However I struggle to see how this will work together with the odb >>> compiler. >> You would normally use native ODB compiler for your development machine. >> So if you are using x86-64 Linux to develop for ARM Android, then you >> would run x86-64 Linux ODB compiler, generate the database support >> source code (the source code is the same for all the platforms), then >> compile that for ARM Android along with your other source code, and >> finally link everything together with cross-compiled libodb*. >> >> It may be possible to build a native ODB compiler for Android but we >> haven't tried (you will most likely need to build GCC from scratch >> with plugin support). > > Hey Boris, > > thx for your clarification - seems the procedure works better when I > use version 17 of the > android ndk. > I'm still in the compilation process, so I can probably report back > once it is finished. > > Given the earlier mentioned errors just disappear when using the 17 > NDK, I wonder how/if it is possible > for the odb compiler (on the host machine) to generate valid c++ > code for the target, when the target probably > uses a completely different standard c/c++ library and has no > support for GCC at all. > >> It may be possible to build a native ODB compiler for Android but we >> haven't tried (you will most likely need to build GCC from scratch >> with plugin support). > I do not want to go this way either if not possbile, sounds like pain to me. > Does that mean you have tried to use odb on a recent android ndk >= 18? > > > Best Regards, > Wolfgang From haupt.wolfgang at gmail.com Tue Apr 14 10:11:27 2020 From: haupt.wolfgang at gmail.com (Wolfgang Haupt) Date: Tue Apr 14 10:21:08 2020 Subject: [odb-users] ODB on Android In-Reply-To: <20200414125659.Horde.VS7JP4VXzisswZbqZShUM-2@webmail.byterefinery.de> References: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> <6f744849-b562-06e9-85c7-5654c4079568@gmail.com> <20200414125659.Horde.VS7JP4VXzisswZbqZShUM-2@webmail.byterefinery.de> Message-ID: On 14.04.20 14:56, c.sell@byterefinery.de wrote: > Hi, > > IMO and according to my experience, the ODB compiler has no dependency > on the target system, including the c/c++ libraries. After all, it is > just a code generator. As long as it generates valid standard C++ > code, it should work on any system. > > I have compiled on ealier NDKs without issues. > > Chris Yeah, that's what I think as well. However, until now I face various low level issues that are hard for me to track down. My run with NDK 17c finished, everything compiled but I'm stuck with a linker error. Not sure what's that about, but seems to be related to the standard library. //home/a1rwulf/Android/omniyon-depends/arm-linux-androideabi-21-release/lib/libodb-sqlite.a(options.o):options.cxx:function std::__ndk1::basic_filebuf >::imbue(std::__ndk1::locale const&): error: undefined reference to 'std::__ndk1::codecvt::id'// ///home/a1rwulf/Android/omniyon-depends/arm-linux-androideabi-21-release/lib/libodb-sqlite.a(options.o):options.cxx:function std::__ndk1::basic_filebuf >::basic_filebuf(): error: undefined reference to 'std::__ndk1::codecvt::id'// ///home/a1rwulf/Android/omniyon-depends/arm-linux-androideabi-21-release/lib/libodb-sqlite.a(options.o):options.cxx:function std::__ndk1::basic_filebuf >::basic_filebuf(): error: undefined reference to 'std::__ndk1::codecvt::id'// //clang60++: error: linker command failed with exit code 1 (use -v to see invocation)// //CMakeFiles/kodi.dir/build.make:538: recipe for target 'libkodi.so' failed// //make[2]: *** [libkodi.so] Error 1// //CMakeFiles/Makefile2:755: recipe for target 'CMakeFiles/kodi.dir/all' failed// //make[1]: *** [CMakeFiles/kodi.dir/all] Error 2// //Makefile:129: recipe for target 'all' failed// //make: *** [all] Error 2/ BR, Wolfgang > > Zitat von Wolfgang Haupt : > >> On 14.04.20 12:02, Boris Kolpackov wrote: >>> Wolfgang Haupt writes: >>> >>>> And I can cross-compile libodb and libodb-sqlite+libodb-mysql just >>>> fine. >>>> >>>> However I struggle to see how this will work together with the odb >>>> compiler. >>> You would normally use native ODB compiler for your development >>> machine. >>> So if you are using x86-64 Linux to develop for ARM Android, then you >>> would run x86-64 Linux ODB compiler, generate the database support >>> source code (the source code is the same for all the platforms), then >>> compile that for ARM Android along with your other source code, and >>> finally link everything together with cross-compiled libodb*. >>> >>> It may be possible to build a native ODB compiler for Android but we >>> haven't tried (you will most likely need to build GCC from scratch >>> with plugin support). >> >> Hey Boris, >> >> thx for your clarification - seems the procedure works better when I >> use version 17 of the >> android ndk. >> I'm still in the compilation process, so I can probably report back >> once it is finished. >> >> Given the earlier mentioned errors just disappear when using the 17 >> NDK, I wonder how/if it is possible >> for the odb compiler (on the host machine) to generate valid c++ code >> for the target, when the target probably >> uses a completely different standard c/c++ library and has no support >> for GCC at all. >> >>> It may be possible to build a native ODB compiler for Android but we >>> haven't tried (you will most likely need to build GCC from scratch >>> with plugin support). >> I do not want to go this way either if not possbile, sounds like pain >> to me. >> Does that mean you have tried to use odb on a recent android ndk >= 18? >> >> >> Best Regards, >> Wolfgang > > > From c.sell at byterefinery.de Tue Apr 14 10:37:19 2020 From: c.sell at byterefinery.de (c.sell@byterefinery.de) Date: Tue Apr 14 10:46:49 2020 Subject: [odb-users] ODB on Android In-Reply-To: References: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> <6f744849-b562-06e9-85c7-5654c4079568@gmail.com> <20200414125659.Horde.VS7JP4VXzisswZbqZShUM-2@webmail.byterefinery.de> Message-ID: <20200414143719.Horde.NKD0UXnjaZDmTw6PzRLrvvO@webmail.byterefinery.de> this is from stackoverflow: Your boost library was built with libc++ (you can tell because it has references to std::__ndk1::* rather than just std::*). You need to use libc++, not gnustl. HTH Zitat von Wolfgang Haupt : > On 14.04.20 14:56, c.sell@byterefinery.de wrote: >> Hi, >> >> IMO and according to my experience, the ODB compiler has no >> dependency on the target system, including the c/c++ libraries. >> After all, it is just a code generator. As long as it generates >> valid standard C++ code, it should work on any system. >> >> I have compiled on ealier NDKs without issues. >> >> Chris > > > Yeah, that's what I think as well. > However, until now I face various low level issues that are hard for > me to track down. > > My run with NDK 17c finished, everything compiled but I'm stuck with > a linker error. > Not sure what's that about, but seems to be related to the standard library. > //home/a1rwulf/Android/omniyon-depends/arm-linux-androideabi-21-release/lib/libodb-sqlite.a(options.o):options.cxx:function std::__ndk1::basic_filebuf >::imbue(std::__ndk1::locale const&): error: undefined reference to 'std::__ndk1::codecvt __mbstate_t>::id'// > ///home/a1rwulf/Android/omniyon-depends/arm-linux-androideabi-21-release/lib/libodb-sqlite.a(options.o):options.cxx:function std::__ndk1::basic_filebuf >::basic_filebuf(): error: undefined reference to 'std::__ndk1::codecvt __mbstate_t>::id'// > ///home/a1rwulf/Android/omniyon-depends/arm-linux-androideabi-21-release/lib/libodb-sqlite.a(options.o):options.cxx:function std::__ndk1::basic_filebuf >::basic_filebuf(): error: undefined reference to 'std::__ndk1::codecvt __mbstate_t>::id'// > //clang60++: error: linker command failed with exit code 1 (use -v > to see invocation)// > //CMakeFiles/kodi.dir/build.make:538: recipe for target 'libkodi.so' failed// > //make[2]: *** [libkodi.so] Error 1// > //CMakeFiles/Makefile2:755: recipe for target > 'CMakeFiles/kodi.dir/all' failed// > //make[1]: *** [CMakeFiles/kodi.dir/all] Error 2// > //Makefile:129: recipe for target 'all' failed// > //make: *** [all] Error 2/ > > BR, > Wolfgang > > >> >> Zitat von Wolfgang Haupt : >> >>> On 14.04.20 12:02, Boris Kolpackov wrote: >>>> Wolfgang Haupt writes: >>>> >>>>> And I can cross-compile libodb and libodb-sqlite+libodb-mysql just fine. >>>>> >>>>> However I struggle to see how this will work together with the >>>>> odb compiler. >>>> You would normally use native ODB compiler for your development machine. >>>> So if you are using x86-64 Linux to develop for ARM Android, then you >>>> would run x86-64 Linux ODB compiler, generate the database support >>>> source code (the source code is the same for all the platforms), then >>>> compile that for ARM Android along with your other source code, and >>>> finally link everything together with cross-compiled libodb*. >>>> >>>> It may be possible to build a native ODB compiler for Android but we >>>> haven't tried (you will most likely need to build GCC from scratch >>>> with plugin support). >>> >>> Hey Boris, >>> >>> thx for your clarification - seems the procedure works better when >>> I use version 17 of the >>> android ndk. >>> I'm still in the compilation process, so I can probably report >>> back once it is finished. >>> >>> Given the earlier mentioned errors just disappear when using the >>> 17 NDK, I wonder how/if it is possible >>> for the odb compiler (on the host machine) to generate valid c++ >>> code for the target, when the target probably >>> uses a completely different standard c/c++ library and has no >>> support for GCC at all. >>> >>>> It may be possible to build a native ODB compiler for Android but we >>>> haven't tried (you will most likely need to build GCC from scratch >>>> with plugin support). >>> I do not want to go this way either if not possbile, sounds like >>> pain to me. >>> Does that mean you have tried to use odb on a recent android ndk >= 18? >>> >>> >>> Best Regards, >>> Wolfgang >> >> >> From boris at codesynthesis.com Tue Apr 14 10:59:47 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Apr 14 11:09:21 2020 Subject: [odb-users] ODB on Android In-Reply-To: <20200414143719.Horde.NKD0UXnjaZDmTw6PzRLrvvO@webmail.byterefinery.de> References: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> <6f744849-b562-06e9-85c7-5654c4079568@gmail.com> <20200414125659.Horde.VS7JP4VXzisswZbqZShUM-2@webmail.byterefinery.de> <20200414143719.Horde.NKD0UXnjaZDmTw6PzRLrvvO@webmail.byterefinery.de> Message-ID: c.sell@byterefinery.de writes: > Your boost library was built with libc++ (you can tell because it has > references to std::__ndk1::* rather than just std::*). You need to use > libc++, not gnustl. Hm, I don't think this is the same problem. > > My run with NDK 17c finished, everything compiled but I'm stuck with a > > linker error. > > > > undefined reference to 'std::__ndk1::codecvt::id' This looks like some kind of misconfiguration. Can you make sure you rebuild all the libodb* libraries for this new NDK (if they were still built with the other version this would explain the error). If this doesn't help, google "undefined reference std::__ndk1::codecvt Android" (without quotes) -- people seem to have this issue with various solutions (upgrade CMake, change project settings, etc). From haupt.wolfgang at gmail.com Tue Apr 14 11:03:49 2020 From: haupt.wolfgang at gmail.com (Wolfgang Haupt) Date: Tue Apr 14 11:13:29 2020 Subject: [odb-users] ODB on Android In-Reply-To: References: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> <6f744849-b562-06e9-85c7-5654c4079568@gmail.com> <20200414125659.Horde.VS7JP4VXzisswZbqZShUM-2@webmail.byterefinery.de> <20200414143719.Horde.NKD0UXnjaZDmTw6PzRLrvvO@webmail.byterefinery.de> Message-ID: <00eedfb8-7037-e107-d660-1bb81e0c0bd7@gmail.com> On 14.04.20 16:59, Boris Kolpackov wrote: > c.sell@byterefinery.de writes: > >> Your boost library was built with libc++ (you can tell because it has >> references to std::__ndk1::* rather than just std::*). You need to use >> libc++, not gnustl. > Hm, I don't think this is the same problem. Yeah it's definitely not the same problem, I don't even use boost. And I only have this 3 errors in a million line project, the project is built against libc++ wich is default in NDK 17. If it was the same issue, I'd probably get thousands of similar errors. > > >>> My run with NDK 17c finished, everything compiled but I'm stuck with a >>> linker error. >>> >>> undefined reference to 'std::__ndk1::codecvt::id' > This looks like some kind of misconfiguration. Can you make sure you > rebuild all the libodb* libraries for this new NDK (if they were > still built with the other version this would explain the error). > > If this doesn't help, google "undefined reference std::__ndk1::codecvt > Android" (without quotes) -- people seem to have this issue with > various solutions (upgrade CMake, change project settings, etc). Yeah I suspected a similar issue, until now I couldn't figure out what's causing it. Googling didn't help so far. I removed all my libs again and rebuilt everything from scratch and still get this, so I'm a bit puzzled. Anyway thanks for your thoughts so far. From boris at codesynthesis.com Wed Apr 15 08:49:04 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Apr 15 08:58:39 2020 Subject: [odb-users] ODB on Android In-Reply-To: <00eedfb8-7037-e107-d660-1bb81e0c0bd7@gmail.com> References: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> <6f744849-b562-06e9-85c7-5654c4079568@gmail.com> <20200414125659.Horde.VS7JP4VXzisswZbqZShUM-2@webmail.byterefinery.de> <20200414143719.Horde.NKD0UXnjaZDmTw6PzRLrvvO@webmail.byterefinery.de> <00eedfb8-7037-e107-d660-1bb81e0c0bd7@gmail.com> Message-ID: Wolfgang Haupt writes: > And I only have this 3 errors in a million line project, the project is > built against libc++ wich is default in NDK 17. > If it was the same issue, I'd probably get thousands of similar errors. It could be that only libodb uses that particular part of libc++ that causes the error (the unresolved symbol actually points into the code from libc++ itself). > I removed all my libs again and rebuilt everything from scratch and still > get this, so I'm a bit puzzled. One thing I would try is take the code from libodb that triggers the error and create a small isolated example from it. Then try to build and link this isolated example. If that works, then it would indicate that libodb is not built in the right configuration for some reason. From icewill9999 at vip.qq.com Tue Apr 14 22:58:40 2020 From: icewill9999 at vip.qq.com (=?gb18030?B?srvD99Xmz+A=?=) Date: Wed Apr 15 08:59:04 2020 Subject: [odb-users] how to check a lazy pointer is null? Message-ID: Hi there, The odb defined header file is like below: #pragma db object class SeismicWavelet{#pragma db inverse(seismicWavelet_) lazy_weak_ptr References: Message-ID: ???? writes: > And my question is how to check the lazy pointer is null? https://www.codesynthesis.com/pipermail/odb-users/2018-March/003963.html BTW, found by punching "lazy pointer null" (without quotes) in this search form: https://www.codesynthesis.com/pipermail/odb-users/ > BTW, i have used ODB for a while, and it's a very great work, i am > getting love it more and more, thanks you guys the ODB developers. Thanks, glad you find it useful. From filyobendeguz at gmail.com Wed Apr 15 10:36:24 2020 From: filyobendeguz at gmail.com (=?UTF-8?B?QmVuZGVnw7p6IEZpbHnDsw==?=) Date: Wed Apr 15 10:46:17 2020 Subject: [odb-users] Compile error Message-ID: Hello I'm currently developing CodeCompass, a software comprehension tool, which uses odb. We noticed that the official ubuntu version of odb is not working. So I tried to dig into that and I compiled odb manually. Unfortunately odb doesn't even compile right now, on Ubuntu 18.04 neither with gcc-7 and gcc-5. From boris at codesynthesis.com Wed Apr 15 11:35:57 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Apr 15 11:45:33 2020 Subject: [odb-users] Compile error In-Reply-To: References: Message-ID: Bendeg?z Fily? writes: > I'm currently developing CodeCompass, a software comprehension tool, which > uses odb. We noticed that the official ubuntu version of odb is not > working. Yes, please upvote this bug report: https://bugs.launchpad.net/bugs/1871095 > So I tried to dig into that and I compiled odb manually. > Unfortunately odb doesn't even compile right now, on Ubuntu 18.04 neither > with gcc-7 and gcc-5. Try the latest pre-release using these instructions: https://codesynthesis.com/products/odb/doc/install-build2.xhtml It builds successfully on our CI with up to GCC 9. From javier.gutierrez at web.de Wed Apr 15 13:59:18 2020 From: javier.gutierrez at web.de (Javier Gutierrez) Date: Wed Apr 15 14:09:04 2020 Subject: [odb-users] Examples for version 2.5 Message-ID: <00a601d6134f$998172b0$cc845810$@web.de> Hi Boris, Time to time I go back to the examples package to try something out. Recently I upgraded to version 2.5. Is there such package for this version as it was for 2.4 (odb-examples-2.4.0.zip) ? Thanks in advance, Javier Gutierrez From boris at codesynthesis.com Thu Apr 16 06:43:32 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Apr 16 06:53:09 2020 Subject: [odb-users] Examples for version 2.5 In-Reply-To: <00a601d6134f$998172b0$cc845810$@web.de> References: <00a601d6134f$998172b0$cc845810$@web.de> Message-ID: Javier Gutierrez writes: > Time to time I go back to the examples package to try something out. > Recently I upgraded to version 2.5. Is there such package for this version > as it was for 2.4 (odb-examples-2.4.0.zip) ? Not yet, but we will add it, hopefully soon. In the meantime, you can use odb-examples-2.4.0 (nothing really changed there). If you are on Windows, see: https://www.codesynthesis.com/pipermail/odb-users/2019-May/004256.html From sam.degheldere at bricsys.com Thu Apr 16 10:02:32 2020 From: sam.degheldere at bricsys.com (sam.degheldere@bricsys.com) Date: Thu Apr 16 10:11:52 2020 Subject: [odb-users] query a container Message-ID: <000001d613f7$ad2807e0$077817a0$@bricsys.com> Hi I have class that trimmed down looks like this: #pragma db object class Commit { public: #pragma db transient std::map> m_files; using file_types = std::map>; #pragma db member(commit_file) \ id_column("commit_id") \ key_column("path") \ value_column("storage_id") \ virtual(file_types) \ get(getUtf8Map(this.m_files)) \ set(fromUtf8Map(this.m_files, (?))) }; I was wondering if I can somehow query on this map? Specifically I'd like to query all my commits that contain a certain path (string/odstring). Is this possible? Thanks in advance Kind regards Sam From boris at codesynthesis.com Fri Apr 17 10:12:34 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Apr 17 10:22:17 2020 Subject: [odb-users] query a container In-Reply-To: <000001d613f7$ad2807e0$077817a0$@bricsys.com> References: <000001d613f7$ad2807e0$077817a0$@bricsys.com> Message-ID: sam.degheldere@bricsys.com writes: > #pragma db object > class Commit > { > std::map> m_files; > } > > I was wondering if I can somehow query on this map? > > Specifically I'd like to query all my commits that contain a certain path > (string/odstring). Currently, the only way to query on containers is using native views (i.e., raw SQL). With one exception: if the container established a relationship (like in your example), then you could query on the pointed-to object using an object view. So, in your example, a native view is the only way to query based on the map key (OdString). But for FileData you can do: #pragma db view object(Commit) object(FileData) struct CommitFileView { std::shared_ptr commit; }; And then: using Query = odb::query; db.query (Query::FileData::path == mypath) From javier.gutierrez at web.de Tue Apr 21 11:52:26 2020 From: javier.gutierrez at web.de (Javier Gutierrez) Date: Tue Apr 21 12:02:36 2020 Subject: [odb-users] Coredump in multithreading environment and one transaction Message-ID: <008c01d617f4$df87db10$9e979130$@web.de> Hi Boris, I have a bunch of threads that share the same transaction. All of them run different queries concurrently. For running the query they call a function passing the transaction and the odb::query by value. The transaction is passed as an object that contains the odb::transaction and a std::mutex. So this function locks the mutex, sets the transaction as odb::current, runs the query and returns the result to the calling thread. Then the thread reads the result in a for-loop. The segmentation fault occurs (if it occurs) when returning the result. Of course when the function returns, the locking of the mutex goes out of scope. Now if I make a test and run the for-loop of the result inside the scope of the mutex then I get no coredump (at least this is the conclusion I got to). But my understanding is that reading the result of the locked area should not be a problem since the result is cached (i.e. the odb::query run with the cache flag = true). Any idea why I might be getting the segmentation fault here? Hope the explanation is clear... Regards, Javier Gutierrez From boris at codesynthesis.com Thu Apr 23 08:09:18 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Apr 23 08:19:18 2020 Subject: [odb-users] Coredump in multithreading environment and one transaction In-Reply-To: <008c01d617f4$df87db10$9e979130$@web.de> References: <008c01d617f4$df87db10$9e979130$@web.de> Message-ID: Javier Gutierrez writes: > I have a bunch of threads that share the same transaction. All of them run > different queries concurrently. > For running the query they call a function passing the transaction and the > odb::query by value. > The transaction is passed as an object that contains the odb::transaction > and a std::mutex. > So this function locks the mutex, sets the transaction as odb::current, runs > the query and returns the result to the calling thread. > Then the thread reads the result in a for-loop. > > The segmentation fault occurs (if it occurs) when returning the result. > > Of course when the function returns, the locking of the mutex goes out of > scope. Now if I make a test and run the for-loop of the result inside the > scope of the mutex then I get no coredump (at least this is the conclusion I > got to). > > But my understanding is that reading the result of the locked area should > not be a problem since the result is cached (i.e. the odb::query run with > the cache flag = true). My guess (without looking at the stack trace) is that this has to do with the object image (essentially, buffers) that are used to load objects from their database representation and are per-connection. In your case they would be shared without any protection. One way to fix this would be to iterate over the result under the lock, load all the objects, and return them as a vector or some such. Note also that query result caching is to be able to have several query results at the same time in a single transaction. Also, for some databases, caching is a noop/unsupported. See the Limitations section for your database in the manual for details. From javier.gutierrez at web.de Thu Apr 23 10:29:58 2020 From: javier.gutierrez at web.de (Javier Gutierrez) Date: Thu Apr 23 10:40:11 2020 Subject: [odb-users] Coredump in multithreading environment and one transaction In-Reply-To: References: <008c01d617f4$df87db10$9e979130$@web.de> Message-ID: <015101d6197b$ade69550$09b3bff0$@web.de> Thanks for the reply. > My guess (without looking at the stack trace) is that this has to do with the > object image (essentially, buffers) that are used to load objects from their > database representation and are per-connection. In your case they would be > shared without any protection. One way to fix this would be to iterate over the > result under the lock, load all the objects, and return them as a vector or some > such. You mean there might be cases where such objects image would be shared with protection ? Do you have any examples so I can have a better understanding of what you mean ? > Note also that query result caching is to be able to have several query results at > the same time in a single transaction. Yes that was my understanding. But I guess the result catching idea gets defeated for a single transaction + multiple threads if you have to use the fix you recommend above. > Also, for some databases, caching is a > noop/unsupported. See the Limitations section for your database in the manual > for details. Hmm I need to support all database then I am better off going for your fix. For completeness here the stack trace: #0 0x00007fac3914feb2 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #1 0x00007fac391624dc in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #2 0x00007fac39162613 in SSL_write () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1 #3 0x00007fac39767fb7 in vio_ssl_write(Vio*, unsigned char const*, unsigned long) () from /usr/local/lib/libmysqlclient-21.so #4 0x00007fac39776d06 in net_write_packet(NET*, unsigned char const*, unsigned long) () from /usr/local/lib/libmysqlclient-21.so #5 0x00007fac39776ed3 in net_flush(NET*) () from /usr/local/lib/libmysqlclient-21.so #6 0x00007fac39777861 in net_write_command(NET*, unsigned char, unsigned char const*, unsigned long, unsigned char const*, unsigned long) () from /usr/local/lib/libmysqlclient-21.so #7 0x00007fac397767d0 in cli_advanced_command () from /usr/local/lib/libmysqlclient-21.so #8 0x00007fac3977f06b in mysql_stmt_prepare () from /usr/local/lib/libmysqlclient-21.so #9 0x00007fac3e07da64 in odb::mysql::statement::init(unsigned long, odb::mysql::statement_kind, odb::mysql::binding const*, bool) () from /usr/local/lib/libodb-mysql-2.5.0-b.17.so #10 0x00007fac3e07dc1d in odb::mysql::statement::statement(odb::mysql::connection&, std::__cxx11::basic_string, std::allocator > const&, odb::mysql::statement_kind, odb::mysql::binding const*, bool) () from /usr/local/lib/libodb-mysql-2.5.0-b.17.so #11 0x00007fac3e07e268 in odb::mysql::select_statement::select_statement(odb::mysql::connection&, std::__cxx11::basic_string, std::allocator > const&, bool, bool, odb::mysql::binding&, odb::mysql::binding&) () from /usr/local/lib/libodb-mysql-2.5.0-b.17.so #12 0x00007fac216e8b23 in odb::access::view_traits_impl::query (q=...) at /home/home/user1/dev/build/database/TestView-odb-mysql.cxx:290 #13 0x00007fac216e8d99 in odb::access::view_traits_impl::query (db=..., q=...) at /home/home/user1/dev/build/database/TestView-odb-mysql.cxx:305 #14 0x00007fac2172f785 in odb::access::view_traits_impl::query (db=..., q=...) at /home/home/user1/dev/inc/database/odb/TestView-odb.ixx:28 #15 0x00007fac21732c0d in odb::database::query_::call > (db=..., q=...) at /usr/local/include/odb/database.txx:489 #16 0x00007fac2173163e in odb::database::query (this=0x55b09dba2240, q=..., cache=true) at /usr/local/include/odb/database.txx:19 #17 0x00007fac21730256 in stm::DbReader::getSelectWithCondition (this=0x55b09dc14400, odbTransactionHandler= std::shared_ptr (use count 2, weak count 0) = {...}, inQuery=...) at /home/user1/dev/inc/DatabaseManager.hpp:362 #18 0x00007fac2172d6ce in stm::TestLookupModule::onEventDetailRecord[abi:cxx11](std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr const&, std::shared_ptr const&) (this=0x55b09dc13f50, transaction=std::shared_ptr (use count 17, weak count 0) = {...}, record=std::shared_ptr (use count 1, weak count 0) = {...}, outputRecordGroup=std::shared_ptr (use count 16, weak count 0) = {...}) at /home/user1/dev/src/TestLookupModule.cpp:163 From haupt.wolfgang at gmail.com Mon Apr 27 04:54:05 2020 From: haupt.wolfgang at gmail.com (Wolfgang Haupt) Date: Mon Apr 27 05:04:27 2020 Subject: [odb-users] ODB on Android In-Reply-To: References: <8c5f736f-a8af-a98a-955b-e41aa6c92663@gmail.com> <6f744849-b562-06e9-85c7-5654c4079568@gmail.com> <20200414125659.Horde.VS7JP4VXzisswZbqZShUM-2@webmail.byterefinery.de> <20200414143719.Horde.NKD0UXnjaZDmTw6PzRLrvvO@webmail.byterefinery.de> Message-ID: <94688063-9522-989f-70ca-258990d5fa4c@gmail.com> On 14.04.20 16:59, Boris Kolpackov wrote: > c.sell@byterefinery.de writes: > >> Your boost library was built with libc++ (you can tell because it has >> references to std::__ndk1::* rather than just std::*). You need to use >> libc++, not gnustl. > Hm, I don't think this is the same problem. > > >>> My run with NDK 17c finished, everything compiled but I'm stuck with a >>> linker error. >>> >>> undefined reference to 'std::__ndk1::codecvt::id' > This looks like some kind of misconfiguration. Can you make sure you > rebuild all the libodb* libraries for this new NDK (if they were > still built with the other version this would explain the error). > > If this doesn't help, google "undefined reference std::__ndk1::codecvt > Android" (without quotes) -- people seem to have this issue with > various solutions (upgrade CMake, change project settings, etc). I started all over again and checked all my makefiles again. Indeed I found a rogue path that mixed systemlibs into the game. I successfully built with NDK 18b now. I'll retry with NDK 20b as well, cause there I got some other strange errors, but I'm not sure if all my fixes have been applied when I attempted the NDK20 build. Thx for you help! From boris at codesynthesis.com Mon Apr 27 08:16:55 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Apr 27 08:27:10 2020 Subject: [odb-users] Coredump in multithreading environment and one transaction In-Reply-To: <015101d6197b$ade69550$09b3bff0$@web.de> References: <008c01d617f4$df87db10$9e979130$@web.de> <015101d6197b$ade69550$09b3bff0$@web.de> Message-ID: Javier Gutierrez writes: > You mean there might be cases where such objects image would be shared with > protection ? Do you have any examples so I can have a better understanding > of what you mean ? Search for statement_cache in libodb-mysql. > For completeness here the stack trace: Your stack trace indicates that the underlying MySQL connection is being used concurrently. Perhaps there is a container in your object which gets loaded as you iterate over the results from multiple threads. From Tschoche.Jonas at ddv-mediengruppe.de Wed Apr 29 12:03:42 2020 From: Tschoche.Jonas at ddv-mediengruppe.de (Tschoche.Jonas@ddv-mediengruppe.de) Date: Thu Apr 30 09:02:01 2020 Subject: [odb-users] Schema Migration not working correctly Message-ID: <5D8FC02A-5DE4-46C2-92D8-231D0D84E76F@ddv-mediengruppe.de> Hello, now i use the schema migration and i have some problems. Tody i rename a field on database. I increase the version and compiled and run it. But it throw a error that the table alredy exist. So i looked in the changelog file and i saw that only a add-table statement was there and no alter-table. I think the reason for that is following: The Version ?1? is not created as model. Only empty, but I don?t know why. When I change a table and increase the version then the table appers as add-table. Why odb don?t generate model for first version? I build with cmake and modified use-odb script for cmake from github: odb_compile(SOURCES FILES ${ODB_HEADERS} DB ${DATABASE_LOWER} GENERATE_QUERY GENERATE_SCHEMA SCHEMA_FORMAT separate STANDARD c++17 PROFILE boost/date-time boost/optional boost/uuid INCLUDE_REGEX "%(.*)/(.+)%$2%" CHANGELOG odb-changelog.log CHANGELOG_DIR ${CMAKE_SOURCE_DIR} FKEYS_DEFERRABLE_MODE not_deferrable DEFAULT_POINTER std::shared_ptr INCLUDE ${JSONCPP_INCLUDE_DIRS} ${Boost_INCLUDE_DIRS} ${Boost_INCLUDE_DIR} ) From boris at codesynthesis.com Thu Apr 30 09:32:53 2020 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Apr 30 09:43:15 2020 Subject: [odb-users] Schema Migration not working correctly In-Reply-To: <5D8FC02A-5DE4-46C2-92D8-231D0D84E76F@ddv-mediengruppe.de> References: <5D8FC02A-5DE4-46C2-92D8-231D0D84E76F@ddv-mediengruppe.de> Message-ID: Tschoche.Jonas@ddv-mediengruppe.de writes: > Tody i rename a field on database. I increase the version and compiled > and run it. But it throw a error that the table alredy exist. You will need to be more specific and precise about what exactly you've done. In fact, if you really did "rename a field on database" as you say, then it would have had no effect on anything ODB-related (ODB sees changes in C++ object model, not in databases). So what you probably mean is that you have renamed a data member in a class. Or maybe you have renamed the class itself. Without knowing exactly what you've done it's hard to say what went wrong. I suggest that you provide a small step-by-step test that reproduces the problem. For each step provide the header file to compile and the ODB command line to compile it. From PStath at jmawireless.com Thu Apr 30 12:40:45 2020 From: PStath at jmawireless.com (Paul Stath) Date: Thu Apr 30 12:51:18 2020 Subject: [odb-users] Schema Migration not working correctly In-Reply-To: References: <5D8FC02A-5DE4-46C2-92D8-231D0D84E76F@ddv-mediengruppe.de> Message-ID: As Boris responded, a small example will go a long way towards allowing others to help. I had a similar experience when I first started using schema migration in my project, maybe you are having the same issue? In the "C++ Object Persistence with ODB" manual in the "Object Model Version and Changelog" section (https://www.codesynthesis.com/products/odb/doc/manual.xhtml#13.1) is a very important paragraph. (capitalization added for emphasis) "The changelog is maintained by the ODB compiler. Specifically, you do not need to make any manual changes to this file. You will, however, need to KEEP IT AROUND from one invocation of the ODB compiler to the next. In other words, the changelog file is BOTH the INPUT and the OUTPUT of the ODB compiler. This, for example, means that if your project's source code is stored in a version control repository, then you will most likely want to store the changelog there as well. If you delete the changelog, then any ability to do schema migration will be lost." If the changelog file is not being version controlled along with your ODB persisted object headers, then the ODB compiler has no idea what the schema looked like in a previous version and will pull all tables/columns in the current version. To get a valid changelog for your previous version (version 1), you will need to check out the sources at that version, run the ODB compiler and SAVE the changelog. Note that this changelog should have: Content to build the schema 1 database Put this saved changelog in the working directory you are using for the development of your next version and run the ODB compiler. Review the updated changelog file and you should see any schema changes for version 2 in: Content to modify schema version 1 to schema version 2 I can tell you that when the tools are used properly, schema migration works very well. --- Paul Stath JMA Wireless pstath@jmawireless.com -----Original Message----- From: Boris Kolpackov Sent: Thursday, April 30, 2020 9:33 AM To: Tschoche.Jonas@ddv-mediengruppe.de Cc: odb-users@codesynthesis.com Subject: Re: [odb-users] Schema Migration not working correctly Tschoche.Jonas@ddv-mediengruppe.de writes: > Tody i rename a field on database. I increase the version and compiled > and run it. But it throw a error that the table alredy exist. You will need to be more specific and precise about what exactly you've done. In fact, if you really did "rename a field on database" as you say, then it would have had no effect on anything ODB-related (ODB sees changes in C++ object model, not in databases). So what you probably mean is that you have renamed a data member in a class. Or maybe you have renamed the class itself. Without knowing exactly what you've done it's hard to say what went wrong. I suggest that you provide a small step-by-step test that reproduces the problem. For each step provide the header file to compile and the ODB command line to compile it.