From boris at codesynthesis.com Tue Nov 1 09:37:32 2011 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Nov 1 09:38:30 2011 Subject: [odb-users] libodb-qt 1.6.1 bugfix released Message-ID: Hi, A new bugfix release for the Qt ODB profile library and ODB compiler binary packages is now available. It fixes an issue with the QByteArray to BLOB mapping that affects Qt versions prior to 4.7. You only need to upgrade if you are using ODB with Qt version earlier than 4.7 and experiencing problems with reading a BLOB to QByteArray. For more information on this issue, refer to the following mailing list thread: http://www.codesynthesis.com/pipermail/odb-users/2011-October/000354.html Note that if you are using one of the ODB compiler binaries (as opposed to building one yourself from source code), then you need to upgrade both libodb-qt and the ODB compiler package for your platform. You can download the new packages from the ODB download page: http://www.codesynthesis.com/products/odb/download.xhtml SHA1 checksums for the files in this release are as follows: 0753972b9c94aab34915e95ab9942b33a316a55c libodb-qt-1.6.1.tar.bz2 ee6a49aa53db1ddabe6e55dde9e2ce97c3348c46 libodb-qt-1.6.1.tar.gz a28d5c63e880cee75844879bc864f48834b3f76e libodb-qt-1.6.1.zip 2ce6d95ec9eb045b0fc7946d67df1512f41af0db odb-1.6.1-x86_64-linux-gnu.tar.bz2 82b74d019c35d7d433159e2eb67eccec6d94c71b odb-1.6.1-i686-linux-gnu.tar.bz2 a00439f207923491df4c0290334384e02236ca09 odb-1.6.1-i686-macosx.tar.bz2 80a1a56d3d8ece59132cfb47a18c313b10286c50 odb-1.6.1-i686-windows.zip Boris From past0004 at stud.fh-kl.de Wed Nov 2 07:56:12 2011 From: past0004 at stud.fh-kl.de (Pascal Stoll) Date: Wed Nov 2 07:56:30 2011 Subject: [odb-users] MySQL BLOB Support Message-ID: Hello, i would like to know if it possible to store medium oder large blobs in mysql? By now i can only store normal blobs (64kb). Did i do something wrong? Best regards From boris at codesynthesis.com Wed Nov 2 09:50:02 2011 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Nov 2 09:51:14 2011 Subject: [odb-users] MySQL BLOB Support In-Reply-To: References: Message-ID: Hi Pascal, Pascal Stoll writes: > i would like to know if it possible to store medium oder large blobs in > mysql? By now i can only store normal blobs (64kb). Did i do something > wrong? By default QByteArray (which I assume is what you are using) is mapped to the MySQL BLOB type. In MySQL, BLOB can store maximum 64Kb. If you want to store more, you will need to map it to either MEDIUMBLOB or LONGBLOB. You can do it on the per-type basis: #pragma db value(QByteArray) type("LONGBLOB") class object { ... QByteArray data_; // Mapped to LONGBLOB. }; Or on the per-member basis: class object { ... #pragma db type("MEDIUMBLOB") QByteArray data_; }; You may also want to check other limitations on the BLOB sizes at the bottom of this page from the MySQL documentation: http://dev.mysql.com/doc/refman/5.1/en/blob.html Boris From past0004 at stud.fh-kl.de Wed Nov 2 10:37:42 2011 From: past0004 at stud.fh-kl.de (Pascal Stoll) Date: Wed Nov 2 10:37:59 2011 Subject: AW: [odb-users] MySQL BLOB Support In-Reply-To: References: , Message-ID: It works. Thank you again. Best regards ________________________________________ Von: Boris Kolpackov [boris@codesynthesis.com] Gesendet: Mittwoch, 2. November 2011 14:50 Bis: Pascal Stoll Cc: odb-users@codesynthesis.com Betreff: Re: [odb-users] MySQL BLOB Support Hi Pascal, Pascal Stoll writes: > i would like to know if it possible to store medium oder large blobs in > mysql? By now i can only store normal blobs (64kb). Did i do something > wrong? By default QByteArray (which I assume is what you are using) is mapped to the MySQL BLOB type. In MySQL, BLOB can store maximum 64Kb. If you want to store more, you will need to map it to either MEDIUMBLOB or LONGBLOB. You can do it on the per-type basis: #pragma db value(QByteArray) type("LONGBLOB") class object { ... QByteArray data_; // Mapped to LONGBLOB. }; Or on the per-member basis: class object { ... #pragma db type("MEDIUMBLOB") QByteArray data_; }; You may also want to check other limitations on the BLOB sizes at the bottom of this page from the MySQL documentation: http://dev.mysql.com/doc/refman/5.1/en/blob.html Boris From chris.richards at yellowfeather.co.uk Wed Nov 2 13:24:37 2011 From: chris.richards at yellowfeather.co.uk (Chris Richards) Date: Wed Nov 2 13:24:50 2011 Subject: [odb-users] Support for std::vector and arrays Message-ID: <4EB17CD5.9060303@yellowfeather.co.uk> Hi, Although ODB supports mapping the std::vector type to the MySQL BLOB and BINARY types it doesn't seem to support std::vector, standard arrays or boost arrays. See the class declaration below: #pragma db object class test { public: #pragma db id int id; #pragma db type("BINARY(16)") std::vector buffer1; // OK #pragma db type("BINARY(16)") std::vector buffer2; // not OK #pragma db type("BINARY(16)") unsigned char buffer3[16]; // not OK #pragma db type("BINARY(16)") boost::array buffer4; // not OK } Is it possible to work around, or add support, for these data types? Thanks, Chris From kfmfe04 at gmail.com Thu Nov 3 03:08:37 2011 From: kfmfe04 at gmail.com (Ken Feng) Date: Thu Nov 3 03:08:44 2011 Subject: [odb-users] Support for Bulk Copy Message-ID: Hi, In my current code, I use Mysql's LOAD DATA INFILE to bulk copy thousands of rows in and out of the database. How would I translate this code into odb? Regards, Ken From boris at codesynthesis.com Thu Nov 3 06:06:25 2011 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 3 06:07:46 2011 Subject: [odb-users] Support for Bulk Copy In-Reply-To: References: Message-ID: Hi Ken, Ken Feng writes: > In my current code, I use Mysql's LOAD DATA INFILE to bulk copy > thousands of rows in and out of the database. > > How would I translate this code into odb? I would first load the file into a list of C++ objects. Then split all the objects into chunks (you can play with the chunk size to find the optimal value). Then start a transaction and use the normal persist() function to store objects in each chunk. Once the whole chunk is handled, I would commit the transaction and repeat with the next chunk. It would actually be interesting to compare the performance numbers of the two approaches. The LOAD DATA INFILE has the advantage of loading data directly into InnoDB without any SQL. But if the input file is very large, the undo log can grow uncontrollably, which will slow things down considerably, as discussed in this article: http://www.mysqlperformanceblog.com/2008/07/03/how-to-load-large-files-safely-into-innodb-with-load-data-infile/ That's the reason why in the above approach I suggested chunking the object list -- at the end of each transaction the undo log will be cleared. On the other hand, the persist() approach does use SQL but it also uses prepared statements, so SQL parsing should not be an issue. ODB also sends all the data in binary form, so there is no need to convert from string representation (or, at least, such conversion can be done on the client side, as opposed to the database server). In contrast, LOAD DATA INFILE will do the conversion on the database server. So as long as you have a fast connection between the client and server (e.g., local UNIX socket or gigabit ethernet), the data path in the ODB approach shouldn't be a problem either. The other thing that you can do to speed up the ODB transaction, is to temporarily disable foreign and unique key checks: using namespace odb::core; database& db = ... connection_ptr c (db.connection ()); c->execute ("SET FOREIGN_KEY_CHECKS = 0"); c->execute ("SET UNIQUE_CHECKS = 0"); for (/*chunk loop*/ { transaction t (c->begin ()); ... t.commit (); } c->execute ("SET UNIQUE_CHECKS = 1"); c->execute ("SET FOREIGN_KEY_CHECKS = 1"); If you use replication, then disabling binary logging may also be a good idea (SET SQL_LOG_BIN = 0;) Boris From boris at codesynthesis.com Thu Nov 3 09:02:44 2011 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 3 09:04:08 2011 Subject: [odb-users] ODB 1.7.0.a1 available Message-ID: Hi, The first alpha version for the upcoming ODB 1.7.0 is now available. The NEWS file entries so far are: * Support for optimistic concurrency. For more information refer to the 'optimistic' example in the odb-examples package. * Support for read-only objects, composite value types, and data members. The new readonly pragma can be used to declare one of these entities as read-only. Constant data members are automatically treated as read-only. For more information, refer to Section 11.1.4 "readonly (object)", Section 11.3.6 "readonly (composite value)", and Section 11.4.10 "readonly (data member)" in the ODB manual. * Support for persistent classes without object ids. Such objects have to be explicitly declared as not having object id and they have limited functionality. For more information, refer to Section 11.1.5 "id", in the ODB manual. * Support for mapping char[N], unsigned char[N], and std::vector to the BLOB (or equivalent) types. For more information, refer to Chapters 13 (for MySQL), 14 (for SQLite), 15 (for PostgreSQL), and 16 (for Oracle) in the ODB manual. * Query result iterator now provides the id() function with allows one to get object id without loading the object. For more information, refer to Section 4.4 "Query Result", in the ODB manual. This pre-release (as well as all the future pre-releases) is available from: http://www.codesynthesis.com/download/odb/pre-release/ The ODB compiler binaries are only available for Windows, GNU/Linux (x86 and x86_64), and Solaris (x86). The SHA1 sums for all the files in this pre-release are provided at the end of the email. Testing and feedback are appreciated. 6834a2e7f77cfdcdeb77767dc838ac7e85692e83 libodb-1.7.0.a1.tar.bz2 cb73a76772eb2eabfa8c18b84d6287fbb3af37fc libodb-1.7.0.a1.tar.gz 1584ac1fa6072dd8def9d9303f438d67204cf07c libodb-1.7.0.a1.zip 59a37f4b0350d4d6b455d694380bf5c0e72ca1a9 libodb-boost-1.7.0.a1.tar.bz2 f5a1bfe86fd3867e11b354c2b9e02479f49f01f1 libodb-boost-1.7.0.a1.tar.gz 1396b2cb5717d2104a9620f8bef05cf84ed84075 libodb-boost-1.7.0.a1.zip 889125a35e70c896bfae3d1ee9bce72f4a25d05d libodb-mysql-1.7.0.a1.tar.bz2 eeaf1df07b9dd02ece6e90181855472b1f8b3071 libodb-mysql-1.7.0.a1.tar.gz a021283a8a8253696dd16c142dbd826aed284be1 libodb-mysql-1.7.0.a1.zip 57e88a8f473b2ad520b9ce7a5d0f692433d6cc98 libodb-pgsql-1.7.0.a1.tar.bz2 46211d3b29a5dcb4585261be3e937288c09bcd99 libodb-pgsql-1.7.0.a1.tar.gz eb1bfef6ef60fef7a1935da14be9c8c5b44ff439 libodb-pgsql-1.7.0.a1.zip 857643398ec6f08a7d6cad6d34282338db6fcb98 libodb-qt-1.7.0.a1.tar.bz2 8d0a9d455db4401bad8bcc8800a12a4c75324ff7 libodb-qt-1.7.0.a1.tar.gz 4e9b7ad1eb71729f6f31a068886e7653a817f87f libodb-qt-1.7.0.a1.zip e31eebd82d3673bb10c7e2013791f07be8ceb714 libodb-sqlite-1.7.0.a1.tar.bz2 2535ca57691ea5f931874f0f21a4cc40282017b1 libodb-sqlite-1.7.0.a1.tar.gz 4c4b0fcf6f85cd81b2a67ce97f9db543701825d3 libodb-sqlite-1.7.0.a1.zip 90dfdfaee47e4a4170276f541d8a6486ddda9e85 libodb-tracer-1.7.0.a1.tar.bz2 0bf0867e630baf56a230115445e865b5df5a276d libodb-tracer-1.7.0.a1.tar.gz 5b9215f1d899698ec6c29a458d74ece4f159b260 libodb-tracer-1.7.0.a1.zip 84f87864c613a29afa00984ac05d95ef0fb2a248 odb-1.7.0.a1-i686-linux-gnu.tar.bz2 7e7773137f61d017479677ab9f7dd3f0081a28b6 odb-1.7.0.a1-i686-solaris.tar.bz2 4f233ae493da702000d26453c4538d0f12f2943d odb-1.7.0.a1-i686-windows.zip 27e527fe5100d0c7e7306fa05152007425b6fda4 odb-1.7.0.a1.tar.bz2 d8148fdf6983fba3962348cdc1178733ac454566 odb-1.7.0.a1.tar.gz 6b149c51d9db30e079f124cabf661bfa77225b6b odb-1.7.0.a1-x86_64-linux-gnu.tar.bz2 338a0d59fd25bd39409ec46d88a80261d3b64a99 odb-1.7.0.a1.zip b31ed9d1f52bce8be7fe31c1dd6dbcbaaf66b0ec odb-examples-1.7.0.a1.tar.bz2 fda46c43db3c3c98938f6a7b1948e00300a01852 odb-examples-1.7.0.a1.tar.gz cc49f10300d439a894e0164e333c54c7dc5df865 odb-examples-1.7.0.a1.zip 9ad054f67e215b8510bbab25393ffd9fbdfb82cd odb-tests-1.7.0.a1.tar.bz2 f4c4de3e0acca54b94c2b0c01cdf4b03f8d98f98 odb-tests-1.7.0.a1.tar.gz fbd0bb72582b0e87e6ebaa9f5c7bb086c1772cbd odb-tests-1.7.0.a1.zip Enjoys, Boris From boris at codesynthesis.com Thu Nov 3 09:08:50 2011 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 3 09:10:16 2011 Subject: [odb-users] Re: Optional PRAGMA_DB (id) In-Reply-To: References: Message-ID: Hi Viacheslav, In just released[1] ODB 1.7.0.a1 we added support for both readonly members and objects without id. So both of our wishes seem to come true. I would appreciate it if you could give these features a try and let us know if everything works well for your use-case. [1] http://codesynthesis.com/pipermail/odb-users/2011-November/000385.html Boris From boris at codesynthesis.com Thu Nov 3 09:13:07 2011 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 3 09:14:33 2011 Subject: [odb-users] Plans for implementing optimistic concurrency In-Reply-To: References: <-2793872452730461468@unknownmsgid> Message-ID: Hi Chris, In just released[1] ODB 1.7.0.a1 we added support for optimistic concurrency using object versioning. We are still working on a chapter in the ODB manual for this feature but the 'optimistic' example in the odb-examples package should give you a pretty good idea about how everything works. I would appreciate it if you could give this a try and let us know if everything works well for your use-case. [1] http://www.codesynthesis.com/pipermail/odb-users/2011-November/000385.html Boris From boris at codesynthesis.com Thu Nov 3 09:18:09 2011 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 3 09:19:33 2011 Subject: [odb-users] Support for std::vector and arrays In-Reply-To: <4EB17CD5.9060303@yellowfeather.co.uk> References: <4EB17CD5.9060303@yellowfeather.co.uk> Message-ID: Hi Chris, Chris Richards writes: > Although ODB supports mapping the std::vector type to the MySQL > BLOB and BINARY types it doesn't seem to support std::vector char>, standard arrays or boost arrays. See the class declaration below: > > [...] > > Is it possible to work around, or add support, for these data types? Yes, in ODB you can map pretty much any C++ type to any database type by providing the odb::value_traits class template specialization. The 'mapping' example in the odb-examples package shows how to do this. Also, for the upcoming 1.7.0 release I have added built-in support for mapping char[N], unsigned char[N], and vector to BLOB types (and added boost::array to the TODO list). This support is also included in the just released 1.7.0.a1, if you would like to give it a try: http://www.codesynthesis.com/pipermail/odb-users/2011-November/000385.html Boris From chris.richards at yellowfeather.co.uk Fri Nov 11 07:24:58 2011 From: chris.richards at yellowfeather.co.uk (Chris Richards) Date: Fri Nov 11 07:25:12 2011 Subject: [odb-users] Plans for implementing optimistic concurrency In-Reply-To: <4EBD13D6.8060307@yellowfeather.co.uk> References: <-2793872452730461468@unknownmsgid> <4EBD13D6.8060307@yellowfeather.co.uk> Message-ID: <4EBD141A.4020100@yellowfeather.co.uk> > *From:* Chris Richards > *Date:* 11 November 2011 12:23 > *To:* Boris Kolpackov > *Subject:* [odb-users] Plans for implementing optimistic concurrency > Hi Boris, > > Thanks for implementing this so quickly, I've just gone through the > optimistic example and looked at the sql log and all looks good. I > will look at integrating with our codebase, but I'm happy to say it is > working. > > BTW I have only tested on Linux and mysql > > Thanks, > Chris > > Boris Kolpackov wrote: > *From:* Boris Kolpackov > *Date:* 03 November 2011 13:13 > *To:* Chris Richards > *CC:* "odb-users@codesynthesis.com" > *Subject:* [odb-users] Plans for implementing optimistic concurrency > Hi Chris, > > In just released[1] ODB 1.7.0.a1 we added support for optimistic > concurrency using object versioning. We are still working on a > chapter in the ODB manual for this feature but the 'optimistic' > example in the odb-examples package should give you a pretty > good idea about how everything works. > > I would appreciate it if you could give this a try and let us know > if everything works well for your use-case. > > [1] > http://www.codesynthesis.com/pipermail/odb-users/2011-November/000385.html > > Boris > *From:* Boris Kolpackov > *Date:* 12 October 2011 12:44 > *To:* Chris Richards > *CC:* "odb-users@codesynthesis.com" > *Subject:* [odb-users] Plans for implementing optimistic concurrency > Hi Chris, > > > It is on our TODO list but we haven't scheduled it yet for any > particular release. Having said that, it shouldn't be too difficult > for us to implement this using a member of an object as a version. > Something like this: > > #pragma db object optimistic(version_) > class person > { > ... > > unsigned long version_; > }; > > The ODB will then check/manage the version and throw an exception if > we try to overwrite a commit with update(). > > Would something like this work for your case? If so, would you be > willing to start using a pre-release if we implemented this (we like > to release new features that someone is already using ;-))? > > Boris > *From:* Chris Richards > *Date:* 11 October 2011 17:19 > *To:* "odb-users@codesynthesis.com" > *Subject:* Plans for implementing optimistic concurrency > Hi, > > In the manual it states that optimistic concurrency will be available > in a future version, are you able to give a rough indication of when > this will be available? Say this year? > > Regards, > Chris From ffelixr at gmail.com Thu Nov 17 11:30:54 2011 From: ffelixr at gmail.com (Fernando Felix-Redondo) Date: Thu Nov 17 11:31:01 2011 Subject: [odb-users] Problems while trying the samples Message-ID: Hello all, This is my first time with ODB so I'm a bit lost. I've installed all the component folowing the INSTALL files and I have tried to run some of the samples with my DB (Postgresql 8.1.23). Unfortunately I always get the following error message when I run them: "unsupported binary format for PostgreSQL date-time SQL types" What could be happening? Kind regards Fernando -- Fernando Felix-Redondo From boris at codesynthesis.com Thu Nov 17 13:44:39 2011 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Nov 17 13:49:24 2011 Subject: [odb-users] Problems while trying the samples In-Reply-To: References: Message-ID: Hi Fernando, Fernando Felix-Redondo writes: > This is my first time with ODB so I'm a bit lost. I've installed all the > component folowing the INSTALL files and I have tried to run some of the > samples with my DB (Postgresql 8.1.23). Unfortunately I always get the > following error message when I run them: > > "unsupported binary format for PostgreSQL date-time SQL types" PostgreSQL supports two representations of the TIME and TIMESTAMP types: 64-bit integer and floating-point. The floating-point representation has a number of drawbacks[1] and, AFAICT, since version 8.4 the integer representation is the default. You seem to be using an older version that is still configured to use the floats. ODB only supports the integer representation and checks that it is talking to a compatible server at the connection establishment time. If the date-time representation is floating-point, it throws an exception. I see two ways how you can resolve this. The easiest is probably to upgrade to a newer version of PostgreSQL (or rebuild this version with the integer representation enabled). Alternatively, if you must use this server, then you can disable the check in the ODB source code (connection.cxx in libodb-pgsql; search for "unsupported binary format"). This will work as long as you don't use the TIME and TIMESTAMP types. [1] http://postgresql.1045698.n5.nabble.com/Integer-datetimes-td1979958.html Boris From ffelixr at gmail.com Fri Nov 18 14:25:49 2011 From: ffelixr at gmail.com (Fernando Felix-Redondo) Date: Fri Nov 18 14:25:57 2011 Subject: [odb-users] Problems while trying the samples In-Reply-To: References: Message-ID: Hi Boris, I've installed a newer Postgresql version (8.4) and now I can run the tests without problem. Thank you very much for your help Regards Fernando On Thu, Nov 17, 2011 at 7:44 PM, Boris Kolpackov wrote: > Hi Fernando, > > Fernando Felix-Redondo writes: > > > This is my first time with ODB so I'm a bit lost. I've installed all the > > component folowing the INSTALL files and I have tried to run some of the > > samples with my DB (Postgresql 8.1.23). Unfortunately I always get the > > following error message when I run them: > > > > "unsupported binary format for PostgreSQL date-time SQL types" > > PostgreSQL supports two representations of the TIME and TIMESTAMP > types: 64-bit integer and floating-point. The floating-point > representation has a number of drawbacks[1] and, AFAICT, since > version 8.4 the integer representation is the default. You seem > to be using an older version that is still configured to use the > floats. > > ODB only supports the integer representation and checks that it > is talking to a compatible server at the connection establishment > time. If the date-time representation is floating-point, it throws > an exception. > > I see two ways how you can resolve this. The easiest is probably to > upgrade to a newer version of PostgreSQL (or rebuild this version > with the integer representation enabled). Alternatively, if you must > use this server, then you can disable the check in the ODB source > code (connection.cxx in libodb-pgsql; search for "unsupported binary > format"). This will work as long as you don't use the TIME and > TIMESTAMP types. > > [1] > http://postgresql.1045698.n5.nabble.com/Integer-datetimes-td1979958.html > > Boris >