From lfrench at globalscape.com  Mon Jun 12 09:28:01 2017
From: lfrench at globalscape.com (Lonnie French)
Date: Mon Jun 12 09:31:08 2017
Subject: [odb-users] Database upgrade issue
Message-ID: <81274861790f4291b46ca72659d0fdc5@GSExchange2.forest.intranet.gs>

Hello,
      I'm running into a database upgrade issue. It's not related to a schema change per se, but some value defaulting that must happen during the upgrade that is giving me problems.

My setup:
I have a multi-table database currently on schema version 5. The database is called Workspaces and the two tables I'm trying to upgrade are Workspace and Participant. Both of these tables have a column Period that was added in version 3. These are the values I'm trying to set.

My Goal:
My intent is during an upgrade of an existing database to set the value of Period for both tables but only after this column has just been added to an existing database row. So, when upgrading from a v1 or v2 database to v5, I want the value of Period for existing Workspace and Participant table entries to be a certain value. If upgrading from a database already on v3 or later I do not want to touch the value of Period.

What I've done so far:
I've added a migration method (1st snippet below) to accomplish this.

The problem I'm having:
When I upgrade only the Workspace table everything works great, but when I add the Participation table change to the upgrade an exception (1: no such column: Participation.type) gets thrown and both tables are empty after the upgrade. The "type" column was added to the Participation table in v4. The definition of the accompanying class is given by the 2nd code snippet below.


>From WorkspacesUpgrade.cpp:

static void migrate_notification_period(odb::database& db)
{
       // Default notification setting for all pre-existing workspaces
       for (Workspace& ws : db.query<Workspace>())
       {
             ws.wi.defaultNotifs.period = static_cast<uint32_t>(Shared::Period::Never);
             db.update(ws);
       }

       // Default notification setting for all pre-existing participants
       for (Participation& part : db.query<Participation>())
       {
             part.notifs.period = static_cast<uint32_t>(Shared::Period::Never);
             db.update(part);
       }
}

static const odb::data_migration_entry<3, 2> migrate_notification_period_entry(&migrate_notification_period);



>From Workspaces.h:

#pragma db object
class Participation
{
public:
       #pragma db id type("BLOB")
       Shared::ParticipationID id;

       std::wstring name;
       std::wstring email;
       #pragma db default(RecipientType::None)
       Shared::RecipientType type = Shared::RecipientType::None;
       Permissions perm;
       Notifications notifs;

       lazy_shared_ptr<Workspace> workspace;
};


I've attached the changelog (Workspaces.xml) for the Workspaces database. If I can provide any further clarification please let me know. Thank you in advance for your help

Regards,
Lonnie French | Software Engineer Manager-EFT
global<http://www.globalscape.com/>scape<http://www.globalscape.com/> | +1 (210) 308-8267 ext 5188 | NYSE MKT:GSB<http://www.globalscape.com/cgi-bin/links.cgi?page=gsb>
*This communication, including attachments, is for the exclusive use of the addressee and may contain proprietary, confidential or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this communication and destroy all copies.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: Workspaces.xml
Type: text/xml
Size: 8195 bytes
Desc: Workspaces.xml
Url : http://codesynthesis.com/pipermail/odb-users/attachments/20170612/a31d1dee/Workspaces.bin
From boris at codesynthesis.com  Mon Jun 12 09:44:37 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Mon Jun 12 09:44:46 2017
Subject: [odb-users] Database upgrade issue
In-Reply-To: <81274861790f4291b46ca72659d0fdc5@GSExchange2.forest.intranet.gs>
References: <81274861790f4291b46ca72659d0fdc5@GSExchange2.forest.intranet.gs>
Message-ID: <boris.20170612153950@codesynthesis.com>

Hi Lonnie,

Lonnie French <lfrench@globalscape.com> writes:

> When I upgrade only the Workspace table everything works great, but when
> I add the Participation table change to the upgrade an exception (1: no
> such column: Participation.type) gets thrown and both tables are empty
> after the upgrade. The "type" column was added to the Participation
> table in v4.

The problem you are having (and a possible solution) is described in
Section 13.4, "Soft Object Model Changes", specifically, the third
paragraph.


> static void migrate_notification_period(odb::database& db)
> {
>        // Default notification setting for all pre-existing workspaces
>        for (Workspace& ws : db.query<Workspace>())
>        {
>              ws.wi.defaultNotifs.period = static_cast<uint32_t>(Shared::Period::Never);
>              db.update(ws);
>        }
> 
>        // Default notification setting for all pre-existing participants
>        for (Participation& part : db.query<Participation>())
>        {
>              part.notifs.period = static_cast<uint32_t>(Shared::Period::Never);
>              db.update(part);
>        }
> }

Seeing that you set period to a fixed value, using the default value
pragma might be a simpler way to accomplish this (which also sidesteps
the above issue).

Boris

From aongeeno at openmailbox.org  Wed Jun 14 03:23:41 2017
From: aongeeno at openmailbox.org (aongeeno@openmailbox.org)
Date: Wed Jun 14 03:24:27 2017
Subject: [odb-users] Avoid copying objects from odb::result
Message-ID: <2951198.rL09eoSxN7@99d44166af322fc16a4774097a5f271d>

Good day! How I can avoid copying objects in this code?

PRAGMA_DB(object)
struct Category
{
    Category() {}
    PRAGMA_DB(member id)
    CategoryID id        = -1;
    QString    name;
private:
    friend class odb::access;
};

/* ............................*/

std::unordered_map<int, std::unique_ptr<Category>> categories;

/* ............................*/

auto result(db_->query<Category>());
for (Category & c : r)
    //categories[c.id] = std::make_unique(c); ???

From il1k3pepsi at gmail.com  Thu Jun 15 03:57:58 2017
From: il1k3pepsi at gmail.com (Henri Schwarz)
Date: Thu Jun 15 05:04:28 2017
Subject: [odb-users] Use SQLite functions to create timestamps by default
Message-ID: <CAD1TrxghkfvPD=fwCAaho5R7==22d-vgiLoGRNmqkpdWPXrCbw@mail.gmail.com>

Hi,

in chapter 14.4.7 of the documentation is an example how to set the default
value of a column to the current timestamp.

#pragma db options("DEFAULT CURRENT_TIMESTAMP()")

  date timestamp_;                    // DEFAULT CURRENT_TIMESTAMP()

I tried this using SQLite but it doesn't seem to work right. At first I had
to remove the parentheses at the end (throws exception on schema creation).
I found out that *date* is a typedef for* long* which is mapped to an
INTEGER but the SQLite documentation states that CURRENT_TIMESTAMP returns
a timestring (TEXT).

If the default value of a column is CURRENT_TIME, CURRENT_DATE or
CURRENT_TIMESTAMP, then the value used in the new row is a text
representation of the current UTC date and/or time. /*...*/ The format for
CURRENT_TIMESTAMP is "YYYY-MM-DD HH:MM:SS".

In the means of SQLite this shouldn't be a problem due to the dynamic
typing.

So with a persistent class member declared as *date* or *long* I end up
with a timestamp in the database that seems to be the positive maximum for
the SQLite integer type (4777546605244151150) since it doesn't change when
I recreate the database and start all over.

Nevertheless when I change the timestamp_ member in the persistent class to
*std::string* or use the pragma *type("TEXT")* the column remains empty
when I create items in the table through ODB.

The CREATE TABLE statement with the *std:string* timestamp_ member looks as
follows.


When I insert items manually (using SQLite Browser) the timestring is put
in the row correctly.

So am I getting something completely wrong here or could this even be a
bug?
From il1k3pepsi at gmail.com  Thu Jun 15 04:00:25 2017
From: il1k3pepsi at gmail.com (Henri Schwarz)
Date: Thu Jun 15 05:04:28 2017
Subject: [odb-users] Re: Use SQLite functions to create timestamps by default
In-Reply-To: <CAD1TrxghkfvPD=fwCAaho5R7==22d-vgiLoGRNmqkpdWPXrCbw@mail.gmail.com>
References: <CAD1TrxghkfvPD=fwCAaho5R7==22d-vgiLoGRNmqkpdWPXrCbw@mail.gmail.com>
Message-ID: <CAD1TrxhH25fZ=Mxgex1m83AuuA5rNPpP1EpJ=NiqTMdP9ngywQ@mail.gmail.com>

Whoops,

forgot to insert the CREATE TABLE statement. Here you go.

CREATE TABLE `connection` (
`id` INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT,
`type` INTEGER NOT NULL,
`timestamp` TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
`typeid` TEXT NOT NULL,
`name` TEXT NOT NULL,
`is_default` INTEGER NOT NULL,
`child` INTEGER,
FOREIGN KEY(`child`) REFERENCES `session`(`id`) DEFERRABLE INITIALLY
DEFERRED
);

Regards.


2017-06-15 9:57 GMT+02:00 Henri Schwarz <il1k3pepsi@gmail.com>:

> Hi,
>
> in chapter 14.4.7 of the documentation is an example how to set the
> default value of a column to the current timestamp.
>
> #pragma db options("DEFAULT CURRENT_TIMESTAMP()")
>
>   date timestamp_;                    // DEFAULT CURRENT_TIMESTAMP()
>
> I tried this using SQLite but it doesn't seem to work right. At first I
> had to remove the parentheses at the end (throws exception on schema
> creation). I found out that *date* is a typedef for* long* which is
> mapped to an INTEGER but the SQLite documentation states that
> CURRENT_TIMESTAMP returns a timestring (TEXT).
>
> If the default value of a column is CURRENT_TIME, CURRENT_DATE or
> CURRENT_TIMESTAMP, then the value used in the new row is a text
> representation of the current UTC date and/or time. /*...*/ The format for
> CURRENT_TIMESTAMP is "YYYY-MM-DD HH:MM:SS".
>
> In the means of SQLite this shouldn't be a problem due to the dynamic
> typing.
>
> So with a persistent class member declared as *date* or *long* I end up
> with a timestamp in the database that seems to be the positive maximum for
> the SQLite integer type (4777546605244151150) since it doesn't change when
> I recreate the database and start all over.
>
> Nevertheless when I change the timestamp_ member in the persistent class
> to *std::string* or use the pragma *type("TEXT")* the column remains
> empty when I create items in the table through ODB.
>
> The CREATE TABLE statement with the *std:string* timestamp_ member looks
> as follows.
>
>
> When I insert items manually (using SQLite Browser) the timestring is put
> in the row correctly.
>
> So am I getting something completely wrong here or could this even be a
> bug?
>
>
>
>
>
>
>
>
>
>
>
From boris at codesynthesis.com  Thu Jun 15 08:09:06 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Thu Jun 15 08:09:15 2017
Subject: [odb-users] Avoid copying objects from odb::result
In-Reply-To: <2951198.rL09eoSxN7@99d44166af322fc16a4774097a5f271d>
References: <2951198.rL09eoSxN7@99d44166af322fc16a4774097a5f271d>
Message-ID: <boris.20170615140804@codesynthesis.com>

aongeeno@openmailbox.org <aongeeno@openmailbox.org> writes:

> How I can avoid copying objects in this code?
> 
> for (Category & c : r)
>     //categories[c.id] = std::make_unique(c); ???

This is described in the manual, specifically, Section 4.4, "Query Result".

Boris

From boris at codesynthesis.com  Thu Jun 15 08:22:34 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Thu Jun 15 08:22:43 2017
Subject: [odb-users] Use SQLite functions to create timestamps by default
In-Reply-To: <CAD1TrxghkfvPD=fwCAaho5R7==22d-vgiLoGRNmqkpdWPXrCbw@mail.gmail.com>
References: <CAD1TrxghkfvPD=fwCAaho5R7==22d-vgiLoGRNmqkpdWPXrCbw@mail.gmail.com>
Message-ID: <boris.20170615141405@codesynthesis.com>

Henri Schwarz <il1k3pepsi@gmail.com> writes:

> #pragma db options("DEFAULT CURRENT_TIMESTAMP()")
> date timestamp_; // DEFAULT CURRENT_TIMESTAMP()
> 
> I tried this using SQLite but it doesn't seem to work right. At first I had
> to remove the parentheses at the end (throws exception on schema creation).
> I found out that *date* is a typedef for* long* which is mapped to an
> INTEGER but the SQLite documentation states that CURRENT_TIMESTAMP returns
> a timestring (TEXT).

The example shows how to use the options pragma. The type used for
timestamp is "a date" not anything specific.


> Nevertheless when I change the timestamp_ member in the persistent class to
> *std::string* [...] the column remains empty when I create items in the
> table through ODB.

For the column to be assigned default value it must be unspecified or
NULL on insertion. Try something like this:

#include <odb/nullable.hxx>

#pragma db options("DEFAULT CURRENT_TIMESTAMP()")
odb::nullable<std::string> timestamp_;

Boris

From aongeeno at openmailbox.org  Fri Jun 16 03:24:43 2017
From: aongeeno at openmailbox.org (aongeeno@openmailbox.org)
Date: Fri Jun 16 03:24:54 2017
Subject: [odb-users] Avoid copying objects from odb::result
In-Reply-To: <boris.20170615140804@codesynthesis.com>
References: <2951198.rL09eoSxN7@99d44166af322fc16a4774097a5f271d>
	<boris.20170615140804@codesynthesis.com>
Message-ID: <4048960.0BnfQY9tXo@99d44166af322fc16a4774097a5f271d>

> aongeeno@openmailbox.org <aongeeno@openmailbox.org> writes:
> 
> > How I can avoid copying objects in this code?
> > 
> > for (Category & c : r)
> >     //categories[c.id] = std::make_unique(c); ???
> 
> This is described in the manual, specifically, Section 4.4, "Query Result".
> 
> Boris
> 

Thank you! I solved my problem:

auto r(db_->query<Category>());
for ( auto it (r.begin()); it != r.end(); ++it)
{
	std::unique_ptr<Category> ptr(it.load());
    categories[it.id()] = std::move(ptr);
}


From aongeeno at openmailbox.org  Fri Jun 16 03:29:26 2017
From: aongeeno at openmailbox.org (aongeeno@openmailbox.org)
Date: Fri Jun 16 03:29:38 2017
Subject: [odb-users] [Feature request] ODB should support CREATE TABLE IF
	NOT EXISTS for SQLite (and, preferably, for other drivers)
In-Reply-To: <boris.20170531170650@codesynthesis.com>
References: <21082477.SB8ZGWT8Bl@99d44166af322fc16a4774097a5f271d>
	<boris.20170531170650@codesynthesis.com>
Message-ID: <1534252.EuyWyAAEAI@99d44166af322fc16a4774097a5f271d>

??????, 31 ?????? 2017 ?. 18:11:00 EEST ?? ????????:
> Hi,
> 
> aongeeno@openmailbox.org <aongeeno@openmailbox.org> writes:
> 
> > Much better will be use supported feature CREATE TABLE IF NOT EXISTS.
> 
> This is not supported by all databases/versions. We could implement
> this for those that support it but we will soon have another problem:
> what if the table exists but from some older version of our application?
> To solve this you can use ODB's database schema evolution support (Chapter
> 13) which also takes care of properly detecting whether the schema does
> not yet exist.
> 
> Boris
> 

Thank you! I solved my problem:

odb::transaction t(db_->begin());
odb::schema_catalog::migrate(*db_);
t.commit();

"As an extra convenience, migrate() will also create the database schema if none exists."

From lfrench at globalscape.com  Fri Jun 16 09:20:25 2017
From: lfrench at globalscape.com (Lonnie French)
Date: Fri Jun 16 10:22:15 2017
Subject: [odb-users] Database upgrade issue
In-Reply-To: <boris.20170612153950@codesynthesis.com>
References: <81274861790f4291b46ca72659d0fdc5@GSExchange2.forest.intranet.gs>
	<boris.20170612153950@codesynthesis.com>
Message-ID: <303aacf4b3094671acf1b81db0f16617@GSExchange2.forest.intranet.gs>

Hello Boris,
    Thank you for the quick response! You are correct; the section you pointed to explained my issue and resolution exactly. I do have a question regarding your second recommendation. As I understand it, I don't believe using the default value pragma will help since this defaults the value for both migrations and new instances to the same value which is not what I am trying to achieve. I wanted to set the value to X if migrating from an older database that pre-dates the addition of the column, but set the value to Y for all new instances of the entry thereafter. Apologies if I misunderstand your intent. Thanks again for your help.

Kind regards,

Lonnie French | Software Engineer Manager-EFT
globalscape | +1 (210) 308-8267 ext 5188 | NYSE MKT:GSB

This communication, including attachments, is for the exclusive use of the addressee and may contain proprietary, confidential or privileged information. If you are not the intended recipient, any use, copying, disclosure, dissemination or distribution is strictly prohibited. If you are not the intended recipient, please notify the sender immediately by return email and delete this communication and destroy all copies.
-----Original Message-----
From: Boris Kolpackov [mailto:boris@codesynthesis.com] 
Sent: Monday, June 12, 2017 8:45 AM
To: Lonnie French <lfrench@globalscape.com>
Cc: odb-users@codesynthesis.com
Subject: Re: [odb-users] Database upgrade issue

Hi Lonnie,

Lonnie French <lfrench@globalscape.com> writes:

> When I upgrade only the Workspace table everything works great, but 
> when I add the Participation table change to the upgrade an exception 
> (1: no such column: Participation.type) gets thrown and both tables 
> are empty after the upgrade. The "type" column was added to the 
> Participation table in v4.

The problem you are having (and a possible solution) is described in Section 13.4, "Soft Object Model Changes", specifically, the third paragraph.


> static void migrate_notification_period(odb::database& db) {
>        // Default notification setting for all pre-existing workspaces
>        for (Workspace& ws : db.query<Workspace>())
>        {
>              ws.wi.defaultNotifs.period = static_cast<uint32_t>(Shared::Period::Never);
>              db.update(ws);
>        }
> 
>        // Default notification setting for all pre-existing participants
>        for (Participation& part : db.query<Participation>())
>        {
>              part.notifs.period = static_cast<uint32_t>(Shared::Period::Never);
>              db.update(part);
>        }
> }

Seeing that you set period to a fixed value, using the default value pragma might be a simpler way to accomplish this (which also sidesteps the above issue).

Boris

From boris at codesynthesis.com  Fri Jun 16 11:46:27 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Fri Jun 16 11:46:36 2017
Subject: [odb-users] Database upgrade issue
In-Reply-To: <303aacf4b3094671acf1b81db0f16617@GSExchange2.forest.intranet.gs>
References: <81274861790f4291b46ca72659d0fdc5@GSExchange2.forest.intranet.gs>
	<boris.20170612153950@codesynthesis.com>
	<303aacf4b3094671acf1b81db0f16617@GSExchange2.forest.intranet.gs>
Message-ID: <boris.20170616174148@codesynthesis.com>

Hi Lonnie,

Lonnie French <lfrench@globalscape.com> writes:

> As I understand it, I don't believe using the default value pragma will help
> since this defaults the value for both migrations and new instances to the
> same value which is not what I am trying to achieve. I wanted to set the
> value to X if migrating from an older database that pre-dates the addition
> of the column, but set the value to Y for all new instances of the entry
> thereafter.

In ODB the only way to trigger a column to get assigned a default value
on the database side is to persist it as NULL. So if you have a data
member like this:

uint32_t period;

ODB will never send it as NULL. So provided you don't insert into this
table by other means, the only time the default value will be used is
during migration.

Boris

From haupt.wolfgang at gmail.com  Fri Jun 16 12:21:06 2017
From: haupt.wolfgang at gmail.com (Wolfgang Haupt)
Date: Fri Jun 16 12:21:30 2017
Subject: [odb-users] mysql crash in multithreaded application
Message-ID: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>

Hi,

I use odb 2.4.0 compiled for multidatabase support for mysql and sqlite.
Currently testing with mysql I encounter a crash in my application when 2
threads access the database at the same time.
I have a singleton that holds the odb::database instance and it also has a
getTransaction method to give transactions to threads.
I don't use a connection_factory, so I assume every thread get's a new
connection and it does not close existing ones, if I got it correct from
the docs.

Backtrace:
https://pastebin.com/70aQx741

Thread 1 and Thread 11 are doing a database query at the same time.
The backtrace shows a heap corruption wihtin mysql.

I found following mysql multithreaded information:
https://stackoverflow.com/questions/26936481/multithreaded-programming-with-libmysql
Does odb do this already?
Do I need to create the create the database connection in the main thread
at the very beginning of the application?


Best Regards,
Wolfgang
From il1k3pepsi at gmail.com  Mon Jun 19 03:55:22 2017
From: il1k3pepsi at gmail.com (Henri Schwarz)
Date: Mon Jun 19 09:28:29 2017
Subject: [odb-users] Use SQLite functions to create timestamps by default
In-Reply-To: <boris.20170615141405@codesynthesis.com>
References: <CAD1TrxghkfvPD=fwCAaho5R7==22d-vgiLoGRNmqkpdWPXrCbw@mail.gmail.com>
	<boris.20170615141405@codesynthesis.com>
Message-ID: <CAD1Trxg96_j11S_05vgXLGQQ-L=wRuGNhBkRhnGB9MOQdoTtbQ@mail.gmail.com>

Hi Boris,

thanks for your advice. I didn't use the nullable type so far and as
expected the class member is now "NULL" in the database but still the
default rule doesn't apply. I'll build a small example around this issue to
investigate what's happening.

Kind regards,

Henri

2017-06-15 14:22 GMT+02:00 Boris Kolpackov <boris@codesynthesis.com>:

> Henri Schwarz <il1k3pepsi@gmail.com> writes:
>
> > #pragma db options("DEFAULT CURRENT_TIMESTAMP()")
> > date timestamp_; // DEFAULT CURRENT_TIMESTAMP()
> >
> > I tried this using SQLite but it doesn't seem to work right. At first I
> had
> > to remove the parentheses at the end (throws exception on schema
> creation).
> > I found out that *date* is a typedef for* long* which is mapped to an
> > INTEGER but the SQLite documentation states that CURRENT_TIMESTAMP
> returns
> > a timestring (TEXT).
>
> The example shows how to use the options pragma. The type used for
> timestamp is "a date" not anything specific.
>
>
> > Nevertheless when I change the timestamp_ member in the persistent class
> to
> > *std::string* [...] the column remains empty when I create items in the
> > table through ODB.
>
> For the column to be assigned default value it must be unspecified or
> NULL on insertion. Try something like this:
>
> #include <odb/nullable.hxx>
>
> #pragma db options("DEFAULT CURRENT_TIMESTAMP()")
> odb::nullable<std::string> timestamp_;
>
> Boris
>
From ronnie.c995 at gmail.com  Mon Jun 19 09:42:00 2017
From: ronnie.c995 at gmail.com (Ronnie Chowdhury)
Date: Mon Jun 19 09:45:30 2017
Subject: [odb-users] Possible bug or missing feature
Message-ID: <5947d4a9.89341c0a.df97b.e491@mx.google.com>

If using boost::posix_time (or boost::gregorian_date::date) in a std::vector<>, the generated code appears not to compile

Example:
#pragma db object 
struct TSData
{
	TSData() = default;
#pragma db id auto
	uint32_t id = 0;
#pragma db type ("INTEGER")
	std::vector<boost::posix_time::ptime> ts; // Fails
	std::vector<float> m_close;
	std::vector<float> m_open;
	std::vector<float> m_high;
	std::vector<float> m_low;
};

Error
H:\ Documents\CPP\include\odb/sqlite/traits.hxx(216): note: No user-defined-conversion operator available that can perform this conversion, or the operator cannot be called
  H:\Documents\CPP\include\odb/sqlite/traits.hxx(214): note: while compiling class template member function 'void odb::sqlite::default_value_traits<T,odb::sqlite::id_integer>::set_image(__int64 &,bool &,T)'

It appears to fail here
    template <typename T, database_type_id ID>
    struct default_value_traits
    {
      typedef T value_type;
      typedef T query_type;
      typedef typename image_traits<T, ID>::image_type image_type;

      static void
      set_value (T& v, const image_type& i, bool is_null)
      {
        if (!is_null)
          v = T (i);
        else
          v = T ();
      }

      static void
      set_image (image_type& i, bool& is_null, T v)
      {
        is_null = false;
        i = image_type (v);  // Fails here when using vector<posix_time::ptime>
      }
    };

This works  
struct TSData
{
	TSData() = default;
#pragma db id auto
	uint32_t id = 0;
#pragma db type ("INTEGER")
	boost::posix_time::ptime ts; // OK but this is not what I need
	std::vector<float> m_close;
	std::vector<float> m_open;
	std::vector<float> m_high;
	std::vector<float> m_low;
};

struct TSData
{
	TSData() = default;
#pragma db id auto
	uint32_t id = 0;
#pragma db type ("INTEGER")
	std::vector<uin64_t> ts; // OK and a work around for now
	std::vector<float> m_close;
	std::vector<float> m_open;
	std::vector<float> m_high;
	std::vector<float> m_low;
};

What is the best way to address this?

Ronnie 

From boris at codesynthesis.com  Mon Jun 19 12:29:52 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Mon Jun 19 12:30:01 2017
Subject: [odb-users] mysql crash in multithreaded application
In-Reply-To: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
References: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
Message-ID: <boris.20170619182207@codesynthesis.com>

Hi Wolfgang,

Wolfgang Haupt <haupt.wolfgang@gmail.com> writes:

> I don't use a connection_factory, so I assume every thread get's a new
> connection and it does not close existing ones, if I got it correct from
> the docs.

Right, you are using the connection pool factory by default which will
recycle the connections. So as long as you don't share connections or
transactions between threads, it should work.


> I found following mysql multithreaded information:
> https://stackoverflow.com/questions/26936481/multithreaded-programming-with-libmysql
> Does odb do this already?

Yes, though the way this is done is pretty crazy (thanks to the way MySQL
does things). See odb/mysql/connection-factory.cxx in libodb-mysql for
the gory details.

One thing that you can try is build the 'common/threads' test from the
odb-tests package. If it works then it is most likely the problem is in
your application.

Boris

From obermann.lukas at gmail.com  Mon Jun 19 12:37:46 2017
From: obermann.lukas at gmail.com (Lukas Obermann)
Date: Mon Jun 19 12:38:01 2017
Subject: [odb-users] mysql crash in multithreaded application
In-Reply-To: <boris.20170619182207@codesynthesis.com>
References: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
	<boris.20170619182207@codesynthesis.com>
Message-ID: <33B7C89F-E399-41FE-9EC3-3E35B2910B7A@gmail.com>

Hi Boris,

may I ask a question related to this.
So, we are using odb::transaction::has_current() to check if there is an active transaction.
If not, a new one is created. Is has_current() considering the thread it currently is in? Means can it be used to determine if the current thread has a transaction already?

Thanks,
Lukas


> Am 19.06.2017 um 18:29 schrieb Boris Kolpackov <boris@codesynthesis.com>:
> 
> Hi Wolfgang,
> 
> Wolfgang Haupt <haupt.wolfgang@gmail.com> writes:
> 
>> I don't use a connection_factory, so I assume every thread get's a new
>> connection and it does not close existing ones, if I got it correct from
>> the docs.
> 
> Right, you are using the connection pool factory by default which will
> recycle the connections. So as long as you don't share connections or
> transactions between threads, it should work.
> 
> 
>> I found following mysql multithreaded information:
>> https://stackoverflow.com/questions/26936481/multithreaded-programming-with-libmysql
>> Does odb do this already?
> 
> Yes, though the way this is done is pretty crazy (thanks to the way MySQL
> does things). See odb/mysql/connection-factory.cxx in libodb-mysql for
> the gory details.
> 
> One thing that you can try is build the 'common/threads' test from the
> odb-tests package. If it works then it is most likely the problem is in
> your application.
> 
> Boris
> 

From boris at codesynthesis.com  Mon Jun 19 12:44:31 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Mon Jun 19 12:44:41 2017
Subject: [odb-users] mysql crash in multithreaded application
In-Reply-To: <33B7C89F-E399-41FE-9EC3-3E35B2910B7A@gmail.com>
References: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
	<boris.20170619182207@codesynthesis.com>
	<33B7C89F-E399-41FE-9EC3-3E35B2910B7A@gmail.com>
Message-ID: <boris.20170619184245@codesynthesis.com>

Lukas Obermann <obermann.lukas@gmail.com> writes:

> Is has_current() considering the thread it currently is in? Means
> can it be used to determine if the current thread has a transaction
> already?

Yes, it can *only* be used to determine if the current thread has an
active transaction (it uses TLS to store the current pointer).

Boris

From boris at codesynthesis.com  Mon Jun 19 12:47:54 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Mon Jun 19 12:48:04 2017
Subject: [odb-users] Possible bug or missing feature
In-Reply-To: <5947d4a9.89341c0a.df97b.e491@mx.google.com>
References: <5947d4a9.89341c0a.df97b.e491@mx.google.com>
Message-ID: <boris.20170619184627@codesynthesis.com>

Ronnie Chowdhury <ronnie.c995@gmail.com> writes:

> #pragma db type ("INTEGER")
> std::vector<boost::posix_time::ptime> ts; // Fails

How do you expect a vector to be mapped into an integer in the database?
Perhaps what you mean is this:

#pragma db value_type ("INTEGER")
std::vector<boost::posix_time::ptime> ts;

Boris

From obermann.lukas at gmail.com  Mon Jun 19 12:54:32 2017
From: obermann.lukas at gmail.com (Lukas Obermann)
Date: Mon Jun 19 12:54:47 2017
Subject: [odb-users] mysql crash in multithreaded application
In-Reply-To: <boris.20170619184245@codesynthesis.com>
References: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
	<boris.20170619182207@codesynthesis.com>
	<33B7C89F-E399-41FE-9EC3-3E35B2910B7A@gmail.com>
	<boris.20170619184245@codesynthesis.com>
Message-ID: <1481D26E-A929-40D9-8A0C-DCE8E990C1A8@gmail.com>

Ok, this is good then.

One more question on session in in multiple threads.
We are creating a session once on init. Should that also be done per thread?

Allow me to share the class I am talking about.
https://github.com/xbmc/xbmc/blob/feature_odb/xbmc/dbwrappers/CommonDatabase.cpp <https://github.com/xbmc/xbmc/blob/feature_odb/xbmc/dbwrappers/CommonDatabase.cpp>
https://github.com/xbmc/xbmc/blob/feature_odb/xbmc/dbwrappers/CommonDatabase.h <https://github.com/xbmc/xbmc/blob/feature_odb/xbmc/dbwrappers/CommonDatabase.h>

All threads use that class to basically get a transaction and getting the odb::database object (which is also only created once and then shared for all).
Queries are then done like

m_cdb.getDB()->query_one...

Can that be the source of our issues?

Thanks,
Lukas


> Am 19.06.2017 um 18:44 schrieb Boris Kolpackov <boris@codesynthesis.com>:
> 
> Lukas Obermann <obermann.lukas@gmail.com> writes:
> 
>> Is has_current() considering the thread it currently is in? Means
>> can it be used to determine if the current thread has a transaction
>> already?
> 
> Yes, it can *only* be used to determine if the current thread has an
> active transaction (it uses TLS to store the current pointer).
> 
> Boris

From boris at codesynthesis.com  Mon Jun 19 13:11:45 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Mon Jun 19 13:11:55 2017
Subject: [odb-users] mysql crash in multithreaded application
In-Reply-To: <1481D26E-A929-40D9-8A0C-DCE8E990C1A8@gmail.com>
References: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
	<boris.20170619182207@codesynthesis.com>
	<33B7C89F-E399-41FE-9EC3-3E35B2910B7A@gmail.com>
	<boris.20170619184245@codesynthesis.com>
	<1481D26E-A929-40D9-8A0C-DCE8E990C1A8@gmail.com>
Message-ID: <boris.20170619190945@codesynthesis.com>

Lukas Obermann <obermann.lukas@gmail.com> writes:

> We are creating a session once on init. Should that also be done
> per thread?

If you are talking about odb::session, then, yes, the default implementation
is not thread-safe and is only meant to be used in a single thread.

Boris

From ronnie.c995 at gmail.com  Mon Jun 19 15:08:00 2017
From: ronnie.c995 at gmail.com (Ronnie Chowdhury)
Date: Mon Jun 19 15:19:37 2017
Subject: [odb-users] FK relationships 
Message-ID: <59482110.069ddf0a.69286.3db3@mx.google.com>

Hi

I?m trying to achieve this simple structure but cannot get my head around this.

A table with 

CREATE TABLE Asset (id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT UNIQUE, asset TEXT NOT NULL UNIQUE PRIMARY KEY);

A timeseries set of tables that have a foreign key into the asset

CREATE TABLE TSData (asset INTEGER NOT NULL PRIMARY KEY, CONSTRAINT asset_fk FOREIGN KEY (asset) REFERENCES Asset (id) DEFERRABLE INITIALLY DEFERRED);

CREATE TABLE TSData_close (object_id INTEGER NOT NULL, "index" INTEGER NOT NULL, value REAL, CONSTRAINT object_id_fk FOREIGN KEY (object_id) REFERENCES TSData (asset) ON DELETE CASCADE);
(repeat similar for the remaining columns for timeseries)

The closest I?ve got is

struct TSData;
#pragma db object 
struct Asset
{
public:
	Asset() = default;
	explicit Asset(const std::string& name) : m_asset(name) {}
	const std::string& asset() const { return m_asset; }
	friend class odb::access;

#pragma db id auto
	int id;
	std::string m_asset;
#pragma db value_not_null inverse(asset_)
	std::vector<std::shared_ptr<TSData>> m_ts; // One asset, references array of timeseries
};
#pragma db object
struct TSData
{
	TSData() = default;
#pragma db id auto 
	int asset_id;
	std::vector<int> ts;
	std::vector<float> m_close;
	std::vector<float> m_open;
	std::vector<float> m_high;
	std::vector<float> m_low;

#pragma db not_null 
	std::shared_ptr<Asset> asset_;
};

But this seems to not save anything in the TSData tables, and the Asset name is not unique ( I can repeated put in multiple names with the same text). 

I tried a few combinations of composite keys but not getting anywhere.

Removing the reference to Asset from TSData, and saving timeseries on it?s own seems to work fine.

What am I missing here please?

Ronnie


From haupt.wolfgang at gmail.com  Mon Jun 19 18:14:03 2017
From: haupt.wolfgang at gmail.com (Wolfgang Haupt)
Date: Mon Jun 19 18:14:26 2017
Subject: [odb-users] mysql crash in multithreaded application
In-Reply-To: <boris.20170619182207@codesynthesis.com>
References: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
	<boris.20170619182207@codesynthesis.com>
Message-ID: <CANdnG_maT7AhtoKMG0jKXnf9o0MPnbg-cKR=8Le_P=c-r1kJyw@mail.gmail.com>

Hi Boris,

regarding:
"So as long as you don't share connections or
transactions between threads, it should work."

Does it mean the default mysql factory is not thread safe?
I had a look at the threads sample and you also share the db instance,
with your worker tasks.
I guess it's meant to be used in that way, right?
If so I think I'm doing it right as I also have only 1 db instance that
is shared with the worker threads.


Best Regards,
Wolfgang

Boris Kolpackov <boris@codesynthesis.com> schrieb am Mo., 19. Juni 2017 um
18:30 Uhr:

> Hi Wolfgang,
>
> Wolfgang Haupt <haupt.wolfgang@gmail.com> writes:
>
> > I don't use a connection_factory, so I assume every thread get's a new
> > connection and it does not close existing ones, if I got it correct from
> > the docs.
>
> Right, you are using the connection pool factory by default which will
> recycle the connections. So as long as you don't share connections or
> transactions between threads, it should work.
>
>
> > I found following mysql multithreaded information:
> >
> https://stackoverflow.com/questions/26936481/multithreaded-programming-with-libmysql
> > Does odb do this already?
>
> Yes, though the way this is done is pretty crazy (thanks to the way MySQL
> does things). See odb/mysql/connection-factory.cxx in libodb-mysql for
> the gory details.
>
> One thing that you can try is build the 'common/threads' test from the
> odb-tests package. If it works then it is most likely the problem is in
> your application.
>
> Boris
>
From tony at rightsoft.com.au  Mon Jun 19 19:01:52 2017
From: tony at rightsoft.com.au (Tony Rietwyk)
Date: Mon Jun 19 19:02:08 2017
Subject: [odb-users] FK relationships 
In-Reply-To: <59482110.069ddf0a.69286.3db3@mx.google.com>
References: <59482110.069ddf0a.69286.3db3@mx.google.com>
Message-ID: <00a401d2e950$0aa61e00$1ff25a00$@rightsoft.com.au>

> Sent: Tuesday, 20 June 2017 5:08 AM
> 
> Hi
> 
> I?m trying to achieve this simple structure but cannot get my head around
> this.
> 
> A table with
> 
> CREATE TABLE Asset (id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT
> UNIQUE, asset TEXT NOT NULL UNIQUE PRIMARY KEY);

That can't possibly work - if you want multiple columns with primary key, then AFAIK you have to use the table primary key syntax.  

But I suspect that don't want it, since the other tables are only referring to Asset(id) in their foreign keys.  In that case, just remove the "PRIMARY KEY" from the Asset.asset field.  

I also think you should have more distinct field names - I would hate to maintain your code where "asset" is a table name, a text field within that table, and an integer field in referring objects!  

Regards, Tony



From boris at codesynthesis.com  Tue Jun 20 08:37:48 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Tue Jun 20 08:37:58 2017
Subject: [odb-users] mysql crash in multithreaded application
In-Reply-To: <CANdnG_maT7AhtoKMG0jKXnf9o0MPnbg-cKR=8Le_P=c-r1kJyw@mail.gmail.com>
References: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
	<boris.20170619182207@codesynthesis.com>
	<CANdnG_maT7AhtoKMG0jKXnf9o0MPnbg-cKR=8Le_P=c-r1kJyw@mail.gmail.com>
Message-ID: <boris.20170620143429@codesynthesis.com>

Wolfgang Haupt <haupt.wolfgang@gmail.com> writes:

> "So as long as you don't share connections or transactions between
> threads, it should work."
> 
> Does it mean the default mysql factory is not thread safe?

No.


> I had a look at the threads sample and you also share the db instance,
> with your worker tasks.

Yes, the database instance can be shared (as in, accessed simultaneously
by multiple threads). Connections and transactions that you get from this
shared database instance cannot.

Boris

From obermann.lukas at gmail.com  Tue Jun 20 09:25:28 2017
From: obermann.lukas at gmail.com (Lukas Obermann)
Date: Tue Jun 20 09:25:43 2017
Subject: [odb-users] mysql crash in multithreaded application
In-Reply-To: <boris.20170620143429@codesynthesis.com>
References: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
	<boris.20170619182207@codesynthesis.com>
	<CANdnG_maT7AhtoKMG0jKXnf9o0MPnbg-cKR=8Le_P=c-r1kJyw@mail.gmail.com>
	<boris.20170620143429@codesynthesis.com>
Message-ID: <A0C452FE-CABA-45B1-A38D-8FDA2DADA659@gmail.com>

Hi Boris,

please let me jump in here again.
So, we are only sharing the database instance, so we should be good.
But, we are still experiencing a large amount of crashes on linux at the statement_cache.

Here is a backtrace. Do you have an idea what can cause this? Can we somehow disable the statement_cache?

Thanks,
Lukas

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fab2971c55a in __GI_abort () at abort.c:89
#2  0x00007fab297584d0 in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7fab29847e68 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007fab2975e956 in malloc_printerr (action=3, str=0x7fab29844b2a "free(): invalid pointer", ptr=<optimized out>, ar_ptr=<optimized out>) at malloc.c:5077
#4  0x00007fab2975f14e in _int_free (av=0x7fab29a7aae0 <main_arena>, p=0x7faae00dce00, have_lock=0) at malloc.c:3873
#5  0x00000000012a06ea in odb::access::object_traits_impl<CODBEpisode, (odb::database_id)0>::image_type::~image_type() ()
#6  0x00000000012a0735 in odb::mysql::view_statements<ODBView_Season_Episodes>::~view_statements() ()
#7  0x00000000012a075d in odb::mysql::view_statements<ODBView_Season_Episodes>::~view_statements() ()
#8  0x00000000012291f0 in std::_Rb_tree<std::type_info const*, std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> >, std::_Select1st<std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> > >, odb::details::type_info_comparator, std::allocator<std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> > > >::_M_erase(std::_Rb_tree_node<std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> > >*) ()
#9  0x00000000012291e3 in std::_Rb_tree<std::type_info const*, std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> >, std::_Select1st<std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> > >, odb::details::type_info_comparator, std::allocator<std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> > > >::_M_erase(std::_Rb_tree_node<std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> > >*) ()
#10 0x0000000001786bb8 in std::_Rb_tree<std::type_info const*, std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> >, std::_Select1st<std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> > >, odb::details::type_info_comparator, std::allocator<std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> > > >::~_Rb_tree() ()
#11 0x0000000001786aea in std::map<std::type_info const*, odb::details::shared_ptr<odb::mysql::statements_base>, odb::details::type_info_comparator, std::allocator<std::pair<std::type_info const* const, odb::details::shared_ptr<odb::mysql::statements_base> > > >::~map() ()
#12 0x0000000001786c42 in odb::mysql::statement_cache::~statement_cache() ()
#13 0x0000000001786c67 in odb::details::unique_ptr<odb::mysql::statement_cache>::~unique_ptr() ()
#14 0x000000000178648d in odb::mysql::connection::~connection() ()
#15 0x000000000176fc24 in odb::mysql::connection_pool_factory::pooled_connection::~pooled_connection() ()
#16 0x000000000176fc40 in odb::mysql::connection_pool_factory::pooled_connection::~pooled_connection() ()
#17 0x0000000000eb79bc in std::vector<odb::details::shared_ptr<odb::mysql::connection_pool_factory::pooled_connection>, std::allocator<odb::details::shared_ptr<odb::mysql::connection_pool_factory::pooled_connection> > >::~vector() ()
#18 0x000000000176e9e0 in odb::mysql::connection_pool_factory::~connection_pool_factory() ()
#19 0x000000000176ea08 in odb::mysql::connection_pool_factory::~connection_pool_factory() ()
#20 0x0000000001772be5 in odb::details::unique_ptr<odb::mysql::connection_factory>::~unique_ptr() ()
#21 0x00000000017706ea in odb::mysql::database::~database() ()
#22 0x000000000177077e in odb::mysql::database::~database() ()
#23 0x0000000000af9e0a in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() ()
#24 0x0000000000cbb1e1 in CVideoDatabase::~CVideoDatabase() ()
#25 0x00000000011cd23e in XFILE::CSmartPlaylistDirectory::GetDirectory(CSmartPlaylist const&, CFileItemList&, std::string const&, bool) ()
#26 0x00000000011ce1bb in XFILE::CSmartPlaylistDirectory::GetDirectory(CURL const&, CFileItemList&) ()
#27 0x00000000011a54e5 in XFILE::CDirectory::GetDirectory(CURL const&, CFileItemList&, XFILE::CDirectory::CHints const&, bool) ()
#28 0x00000000011a5999 in XFILE::CDirectory::GetDirectory(std::string const&, CFileItemList&, XFILE::CDirectory::CHints const&, bool) ()
#29 0x00000000011a5a08 in XFILE::CDirectory::GetDirectory(std::string const&, CFileItemList&, std::string const&, int, bool) ()
#30 0x0000000000e02fc5 in CDirectoryJob::DoWork() ()
#31 0x0000000000d751aa in CJobWorker::Process() ()
#32 0x0000000000dc4a11 in CThread::Action() ()
#33 0x0000000000dc50bd in CThread::staticThread(void*) ()
#34 0x00007fab2def8494 in start_thread (arg=0x7faafaffd700) at pthread_create.c:456
#35 0x00007fab297cf1bf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

> Am 20.06.2017 um 14:37 schrieb Boris Kolpackov <boris@codesynthesis.com>:
> 
> Wolfgang Haupt <haupt.wolfgang@gmail.com> writes:
> 
>> "So as long as you don't share connections or transactions between
>> threads, it should work."
>> 
>> Does it mean the default mysql factory is not thread safe?
> 
> No.
> 
> 
>> I had a look at the threads sample and you also share the db instance,
>> with your worker tasks.
> 
> Yes, the database instance can be shared (as in, accessed simultaneously
> by multiple threads). Connections and transactions that you get from this
> shared database instance cannot.
> 
> Boris
> 


From haupt.wolfgang at gmail.com  Tue Jun 20 17:07:49 2017
From: haupt.wolfgang at gmail.com (Wolfgang Haupt)
Date: Tue Jun 20 17:08:13 2017
Subject: [odb-users] mysql crash in multithreaded application
In-Reply-To: <boris.20170620143429@codesynthesis.com>
References: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
	<boris.20170619182207@codesynthesis.com>
	<CANdnG_maT7AhtoKMG0jKXnf9o0MPnbg-cKR=8Le_P=c-r1kJyw@mail.gmail.com>
	<boris.20170620143429@codesynthesis.com>
Message-ID: <CANdnG_mPEzQGeJCHCono+MKycCs+geVbd=rPu3cESzMLopt_tQ@mail.gmail.com>

Hi,

I've been analysing some backtraces too.
What I found is following:

Thread #1:
Thread 1 (Thread 0x7f8dad5a6700 (LWP 2584)):
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f8daf62a55a in __GI_abort () at abort.c:89
#2  0x00007f8daf6664d0 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7f8daf755e68 "*** Error in `%s': %s: 0x%s ***\n") at
../sysdeps/posix/libc_fatal.c:175
#3  0x00007f8daf66c956 in malloc_printerr (action=3, str=0x7f8daf752adb
"corrupted double-linked list", ptr=<optimized out>, ar_ptr=<optimized
out>) at malloc.c:5077
#4  0x00007f8daf66ccdc in malloc_consolidate (av=av@entry=0x7f8d9c000020)
at malloc.c:4190
#5  0x00007f8daf66e69a in _int_malloc (av=av@entry=0x7f8d9c000020,
bytes=bytes@entry=3400) at malloc.c:3450
#6  0x00007f8daf670664 in __GI___libc_malloc (bytes=3400) at malloc.c:2890
#7  0x00007f8daff44958 in operator new(unsigned long) () from
/usr/lib/libstdc++.so.6
#8  0x00000000012ba51a in ?? ()
#9  0x00000000012cd844 in
odb::mysql::view_statements<ODBView_TVShow_Seasons>&
odb::mysql::statement_cache::find_view<ODBView_TVShow_Seasons>() ()
#10 0x00000000012c4863 in
odb::access::view_traits_impl<ODBView_TVShow_Seasons,*
(odb::database_id)0*>::query(odb::database&,
odb::mysql::query_base const&) ()
#11 0x00000000012c4abb in
odb::access::view_traits_impl<ODBView_TVShow_Seasons,
*(odb::database_id)0*>::query(odb::database&,
odb::query_base const&) ()
#12 0x0000000000d161b3 in bool
odb::database::query_one_<ODBView_TVShow_Seasons, *(odb::database_id)5*,
odb::query<ODBView_TVShow_Seasons, odb::query_base>
>(odb::query<ODBView_TVShow_Seasons, odb::query_base> const&,
ODBView_TVShow_Seasons&) ()
#13 0x0000000000cdf28c in
CVideoDatabase::GetDetailsForEpisode(odb::result_iterator<ODBView_Episode,
(odb::class_kind)1>, int) ()

Thread #9:
Thread 9 (Thread 0x7f8d91ffc700 (LWP 2596)):
#0  0x00007f8db3e0ff0f in __libc_recv (fd=79, buf=0x7f8d7809f7b0, n=16384,
flags=0) at ../sysdeps/unix/sysv/linux/x86_64/recv.c:28
#1  0x0000000001653240 in vio_read ()
#2  0x00000000016532c4 in vio_read_buff ()
#3  0x00000000016519c3 in ?? ()
#4  0x00000000016666a3 in ?? ()
#5  0x0000000001669b3d in my_net_read ()
#6  0x0000000001671168 in cli_safe_read_with_ok ()
#7  0x0000000001674206 in ?? ()
#8  0x0000000001675d1c in ?? ()
#9  0x0000000001675fc3 in cli_stmt_execute ()
#10 0x0000000001675411 in mysql_stmt_execute ()
#11 0x000000000177ff11 in odb::mysql::select_statement::execute() ()
#12 0x00000000012c9591 in odb::access::object_traits_impl<CODBTVShow,*
(odb::database_id)0*>::artwork_traits::load(std::vector<odb::lazy_shared_ptr<CODBArt>,
std::allocator<odb::lazy_shared_ptr<CODBArt> > >&,
odb::mysql::container_statements<odb::access::object_traits_impl<CODBTVShow,
(odb::database_id)0>::artwork_traits>&) ()
#13 0x00000000012c9bf5 in odb::access::object_traits_impl<CODBTVShow,
*(odb::database_id)0*>::section_foreign_traits::load(odb::access::object_traits_impl<CODBTVShow,
(odb::database_id)0>::extra_statement_cache_type&, CODBTVShow&) ()
#14 0x00000000012c9dd7 in odb::access::object_traits_impl<CODBTVShow,
*(odb::database_id)0*>::load(odb::connection&, CODBTVShow&, odb::section&)
()
#15 0x0000000000cb9d38 in ?? ()
#16 0x0000000000cd8582 in CVideoDatabase::GetArtForItem(int, std::string
const&, std::map<std::string, std::string, std::less<std::string>,
std::allocator<std::pair<std::string const, std::string> > >&) ()
#17 0x0000000000d3ae71 in CVideoThumbLoader::FillLibraryArt(CFileItem&) ()
#18 0x0000000000d3a0a4 in CVideoThumbLoader::LoadItemCached(CFileItem*) ()
#19 0x0000000000d39692 in CVideoThumbLoader::LoadItem(CFileItem*) ()
#20 0x0000000000e05300 in CDirectoryJob::DoWork() ()
#21 0x0000000000d771ce in CJobWorker::Process() ()
#22 0x0000000000dc6a35 in CThread::Action() ()
#23 0x0000000000dc70e1 in CThread::staticThread(void*) ()
#24 0x00007f8db3e06494 in start_thread (arg=0x7f8d91ffc700) at
pthread_create.c:456
#25 0x00007f8daf6dd1bf in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97


What is this database_id?
Does it indicate that Thread 9 is overwriting something in Thread 1?

Thanks for any hints.


Best Regards,
Wolfgang

Boris Kolpackov <boris@codesynthesis.com> schrieb am Di., 20. Juni 2017 um
14:37 Uhr:

> Wolfgang Haupt <haupt.wolfgang@gmail.com> writes:
>
> > "So as long as you don't share connections or transactions between
> > threads, it should work."
> >
> > Does it mean the default mysql factory is not thread safe?
>
> No.
>
>
> > I had a look at the threads sample and you also share the db instance,
> > with your worker tasks.
>
> Yes, the database instance can be shared (as in, accessed simultaneously
> by multiple threads). Connections and transactions that you get from this
> shared database instance cannot.
>
> Boris
>
From il1k3pepsi at gmail.com  Tue Jun 20 10:14:07 2017
From: il1k3pepsi at gmail.com (Henri Schwarz)
Date: Wed Jun 21 05:52:53 2017
Subject: [odb-users] Mapping to nullptr_t in templated class interface
Message-ID: <CAD1TrxiMfAB1A1jcD2E4xh5ee0vD2t1YE55Y0zTsnKTVmodSXg@mail.gmail.com>

Hi Boris,

My data model defines several persistent classes which are ordered
hierarchically using 1-1 relationships. To model these relationships I
introduced two templated interface classes that look like this.

template <class T>
struct parent_interface {

using parent_type = T;

// accessor

parent_type const& get_parent() const {

return parent_;

}

// modifier
void set_parent(parent_type const &parent) {

parent_ = parent;

}


protected:

~parent_interface() {}


private:

parent_type parent_;

}


The other interface class is named child_interface and has an equivalent
implementation.
For the example I' ll call the hierarchical persistent classes class
Level_N, e.g. class Level_1, class Level_2, class Level_3 and so forth.

So now it's obvious that the class on the first Level has no parent and the
last one has no child but nevertheless I'd like all persistent classes to
stick to the two interfaces. Therefore I provided definitions like the
following.

#pragma db object
class Level_1 : public parent_interface<nullptr_t>, public
child_interface<Level_2> {
public:

//

// ... ctor, dtor, public methods
//

protected:

friend class odb::access;

private:

//

// ... some data members
//

#pragma db member(parent) virtual(parent_type)

#pragma db member(child) virtual(child_type)

}

#pragma db object
class Level_2 : public parent_interface<Level_1>, public
child_interface<Level_2> {

// ...

}

#pragma db object
class Level_3 : public parent_interface<Level_2>, public
child_interface<nullptr_t> {

// ...

}

So what I'm trying to achieve is a mapping of the parent of the Level_1
persistent class to a null pointer or something similar so that I can do
something like this

if (!get_parent()) { // do something }

So that the start and end points in my hierarchy would be marked with null
pointers.

But the odb compiler gives me an error "unable to map C++ type
'::parent_interface< ::std::nullptr_t >::parent_type' used in data member
'parent' to a SQLite database type". It also states to use "#pragma db
type" to specify the database type but all types I tried so far
also failed. Therefore I'd like to ask you if there is a way to implement
my design or circumvent this compiler error. If it's not possible I'll have
to think of something new.

Your advice is much appreciated.

Kind regards,

Henri

P.S.: The code compiles when I omit the parent_interface inheritance in the
top level class and the child_interface inheritance in the last class of
the hierarchy but that feels more like a hack than like clean code and a
sound design.
From boris at codesynthesis.com  Wed Jun 21 13:15:43 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Wed Jun 21 13:15:51 2017
Subject: [odb-users] Mapping to nullptr_t in templated class interface
In-Reply-To: <CAD1TrxiMfAB1A1jcD2E4xh5ee0vD2t1YE55Y0zTsnKTVmodSXg@mail.gmail.com>
References: <CAD1TrxiMfAB1A1jcD2E4xh5ee0vD2t1YE55Y0zTsnKTVmodSXg@mail.gmail.com>
Message-ID: <boris.20170621191339@codesynthesis.com>

Henri Schwarz <il1k3pepsi@gmail.com> writes:

> But the odb compiler gives me an error "unable to map C++ type
> '::parent_interface< ::std::nullptr_t >::parent_type' used in data member
> 'parent' to a SQLite database type".

You could mark the parent member in Level1 class (that derives from the
nullptr'ed parent_interface) as transient. Probably no use storing it in
the database since it is always NULL.

Boris

From boris at codesynthesis.com  Wed Jun 21 13:23:13 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Wed Jun 21 13:23:22 2017
Subject: [odb-users] mysql crash in multithreaded application
In-Reply-To: <A0C452FE-CABA-45B1-A38D-8FDA2DADA659@gmail.com>
References: <CANdnG_mJpE7C=rjKnnWmKrY5JgAmv0K49y_1svS1aUnBq47L1A@mail.gmail.com>
	<boris.20170619182207@codesynthesis.com>
	<CANdnG_maT7AhtoKMG0jKXnf9o0MPnbg-cKR=8Le_P=c-r1kJyw@mail.gmail.com>
	<boris.20170620143429@codesynthesis.com>
	<A0C452FE-CABA-45B1-A38D-8FDA2DADA659@gmail.com>
Message-ID: <boris.20170621191911@codesynthesis.com>

Lukas Obermann <obermann.lukas@gmail.com> writes:

> Here is a backtrace.

It shows that things go badly when trying to free some memory while
destroying the database instance. This would normally indicate memory
corruption. You can try running your application under valgrind and
see if it shows anything.

Another thought: are you sure the database instance is not destroyed
while there are still connections/transactions used by other threads?


> Do you have an idea what can cause this? Can we somehow disable the
> statement_cache?

No, it is an integral part of ODB.

Boris

From anton.paymyshev at gmail.com  Thu Jun 22 02:30:27 2017
From: anton.paymyshev at gmail.com (Anton Paymyshev)
Date: Thu Jun 22 02:31:00 2017
Subject: [odb-users] Migration issues with on_delete(set_null) constraint
Message-ID: <CADSrH9fAxC+FCy9vHhFAZmNwmz8sRzb5DYfEyCc_+OQxF8Ftxw@mail.gmail.com>

Hello,

My configuration is:
odb 2.5.0.b3
windows 10

In short: ODB compiler does not generate proper update query when a new
field added to db object with on_delete(set_null) constraint.


[See TEST.h at the bottom]

1) odb-compile TEST.h (odb.exe --database sqlite --generate-schema TEST.h)
2) uncomment 3 lines in TEST.h (effectively adds ptr field to struct and
updates schema version)
3) odb-compile TEST.h again
4) look into TEST-odb.cxx:
There is odb::create_schema(...) method which creates table S from scratch
and it adds ON DELETE SET NULL constraint on ptr field:
          db.execute ("CREATE TABLE \"S\" (\n"
                      "  \"id\" INTEGER NOT NULL PRIMARY KEY
AUTOINCREMENT,\n"
                      "  \"ptr\" INTEGER NULL,\n"
                      "  CONSTRAINT \"ptr_fk\"\n"
                      "    FOREIGN KEY (\"ptr\")\n"
                      "    REFERENCES \"S\" (\"id\")\n"
                      "    ON DELETE SET NULL\n"
                      "    DEFERRABLE INITIALLY DEFERRED)");

There is also odb::migrate_schema_2(...) method which adds ptr column but
***NO*** ON DELETE SET NULL constraint:
          db.execute ("ALTER TABLE \"S\"\n"
                      "  ADD COLUMN \"ptr\" INTEGER NULL CONSTRAINT
\"ptr_fk\" REFERENCES \"S\" (\"id\")");

Also migrate query lacks DEFERRABLE INITIALLY DEFERRED constraint which is
not the case for us but it looks like the symptom of same issue.



----TEST.h----
#pragma db model version(1, 1)
//#pragma db model version(1, 2)

#pragma db object
struct S
{
#pragma db id auto
int id;

//#pragma db added(2) on_delete(set_null)
// S* ptr;
};
-------------------
From boris at codesynthesis.com  Thu Jun 22 09:58:38 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Thu Jun 22 09:58:47 2017
Subject: [odb-users] Migration issues with on_delete(set_null) constraint
In-Reply-To: <CADSrH9fAxC+FCy9vHhFAZmNwmz8sRzb5DYfEyCc_+OQxF8Ftxw@mail.gmail.com>
References: <CADSrH9fAxC+FCy9vHhFAZmNwmz8sRzb5DYfEyCc_+OQxF8Ftxw@mail.gmail.com>
Message-ID: <boris.20170622155253@codesynthesis.com>

Anton Paymyshev <anton.paymyshev@gmail.com> writes:

> There is also odb::migrate_schema_2(...) method which adds ptr column but
> ***NO*** ON DELETE SET NULL constraint:
>           db.execute ("ALTER TABLE \"S\"\n"
>                       "  ADD COLUMN \"ptr\" INTEGER NULL CONSTRAINT
> \"ptr_fk\" REFERENCES \"S\" (\"id\")");

Thanks for the test, I could reproduce the problem.

SQLite has limited ALTER TABLE support. Specifically, a constraint
cannot be added by itself, only as part of the column. So we have
to hack around this but apparently we didn't hack far enough. I've
now fixed it:

http://scm.codesynthesis.com/?p=odb/odb.git;a=commit;h=e783a1120b174bac40697000d218796ddf421dfe

The patch is pretty simple and should apply cleanly to previous versions.
Alternatively, I can package another pre-release.

Boris

From anton.paymyshev at gmail.com  Fri Jun 23 05:18:41 2017
From: anton.paymyshev at gmail.com (Anton Paymyshev)
Date: Fri Jun 23 05:19:13 2017
Subject: [odb-users] Migration issues with on_delete(set_null) constraint
In-Reply-To: <boris.20170622155253@codesynthesis.com>
References: <CADSrH9fAxC+FCy9vHhFAZmNwmz8sRzb5DYfEyCc_+OQxF8Ftxw@mail.gmail.com>
	<boris.20170622155253@codesynthesis.com>
Message-ID: <CADSrH9eC+-ZZeTkNEGvBDo90wOjA9-kmAhAJ2wB685FASBt=fA@mail.gmail.com>

Boris, thank you for quick response!

Could you please make new package so I can test it?

On Thu, Jun 22, 2017 at 8:58 PM, Boris Kolpackov <boris@codesynthesis.com>
wrote:

> Anton Paymyshev <anton.paymyshev@gmail.com> writes:
>
> > There is also odb::migrate_schema_2(...) method which adds ptr column but
> > ***NO*** ON DELETE SET NULL constraint:
> >           db.execute ("ALTER TABLE \"S\"\n"
> >                       "  ADD COLUMN \"ptr\" INTEGER NULL CONSTRAINT
> > \"ptr_fk\" REFERENCES \"S\" (\"id\")");
>
> Thanks for the test, I could reproduce the problem.
>
> SQLite has limited ALTER TABLE support. Specifically, a constraint
> cannot be added by itself, only as part of the column. So we have
> to hack around this but apparently we didn't hack far enough. I've
> now fixed it:
>
> http://scm.codesynthesis.com/?p=odb/odb.git;a=commit;h=
> e783a1120b174bac40697000d218796ddf421dfe
>
> The patch is pretty simple and should apply cleanly to previous versions.
> Alternatively, I can package another pre-release.
>
> Boris
>
From boris at codesynthesis.com  Mon Jun 26 12:43:29 2017
From: boris at codesynthesis.com (Boris Kolpackov)
Date: Mon Jun 26 12:43:38 2017
Subject: [odb-users] Migration issues with on_delete(set_null) constraint
In-Reply-To: <CADSrH9eC+-ZZeTkNEGvBDo90wOjA9-kmAhAJ2wB685FASBt=fA@mail.gmail.com>
References: <CADSrH9fAxC+FCy9vHhFAZmNwmz8sRzb5DYfEyCc_+OQxF8Ftxw@mail.gmail.com>
	<boris.20170622155253@codesynthesis.com>
	<CADSrH9eC+-ZZeTkNEGvBDo90wOjA9-kmAhAJ2wB685FASBt=fA@mail.gmail.com>
Message-ID: <boris.20170626184036@codesynthesis.com>

Anton Paymyshev <anton.paymyshev@gmail.com> writes:

> Could you please make new package so I can test it?

Seeing that this is a minor patch that doesn't affect any interfaces,
I've replaced the ODB compiler packages for 2.5.0-b.3 in-place. This
means all you need to do is get the new ODB compiler binary/package
and regenerate your code (no ODB runtime library update is necessary):

http://codesynthesis.com/~boris/tmp/odb/pre-release/b.3/

Let me know if there are any issues.

Boris

From il1k3pepsi at gmail.com  Tue Jun 27 04:42:07 2017
From: il1k3pepsi at gmail.com (Henri Schwarz)
Date: Tue Jun 27 05:24:10 2017
Subject: [odb-users] Mapping to nullptr_t in templated class interface
In-Reply-To: <boris.20170621191339@codesynthesis.com>
References: <CAD1TrxiMfAB1A1jcD2E4xh5ee0vD2t1YE55Y0zTsnKTVmodSXg@mail.gmail.com>
	<boris.20170621191339@codesynthesis.com>
Message-ID: <CAD1TrxiAWh6WnMdpC9qoHmh=ESj5jnrX=i+stFEdQt=yF4oESA@mail.gmail.com>

Hi Boris,

I've solved the problem a little bit different but the essential hint was
marking the data member as transient. Within the interface class I'm using
a type alias to specify the type of the data member.

*template <typename T> *
*struct parent_interface {*

*using parent_type = std::shared_ptr<T>;*
*...*

*}*

Maybe I should rename the class to parent_policy or something, because the
getter and setter methods are implemented right in the place and the data
member belongs to the classes private members.

*{*

*...*

*parent_type const &get_parent() const {*

*return parent_;*



*}*

*void set_parent(parent_type const &parent) {*

*parent_ = parent;*





*}private:*



*#pragma db default(null)parent_type parent_;*

*}*

What did the trick was a full template specialization for the nullptr'ed
case where the data member is marked as transient.

*template <>*
*struct parent_interface<nullptr_t> {*

*...*

*private:*

*#pragma db transient*

*parent_type parent_ {nullptr};*

*}*

Now I can use the (badly named) interface classes in my persistent class
hierarchy. As a bonus the odb compiler produces less code compared to the
case where the interface classes only declare the getter and setter methods
as virtual and the inheriting classes override them.

class Level_1 : public parent_interface<nullptr_t>, public
child_interface<Level_2> { ... }  // compiles now

Kind regards,

Henri



2017-06-21 19:15 GMT+02:00 Boris Kolpackov <boris@codesynthesis.com>:

> Henri Schwarz <il1k3pepsi@gmail.com> writes:
>
> > But the odb compiler gives me an error "unable to map C++ type
> > '::parent_interface< ::std::nullptr_t >::parent_type' used in data member
> > 'parent' to a SQLite database type".
>
> You could mark the parent member in Level1 class (that derives from the
> nullptr'ed parent_interface) as transient. Probably no use storing it in
> the database since it is always NULL.
>
> Boris
>