From christian at gsvitec.com Thu Oct 2 15:02:30 2014 From: christian at gsvitec.com (Christian Sell) Date: Thu Oct 2 15:02:33 2014 Subject: [xsd-users] OutOfMemoryException when loading binary schema Message-ID: <1803753366.1714497.1412276550254.open-xchange@omgreatgod.store> Hello, I have successfully implemented the embedded schema procedure as given in the XSD examples under Win32. Now I have ported everything over to Linux-gcc64, and got the compile to succeed (always with newest Xerces and XSD). However, when I invoke the program, I get a "xercesc_3_1::OutOfMemoryException". When debugging, I see the error happening ate the following statement: gp->deserializeGrammars(&is); Does anyone have a hint what I may be doing wrong? thanks, Christian From boris at codesynthesis.com Fri Oct 3 02:15:40 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Oct 3 02:22:50 2014 Subject: [xsd-users] OutOfMemoryException when loading binary schema In-Reply-To: <1803753366.1714497.1412276550254.open-xchange@omgreatgod.store> References: <1803753366.1714497.1412276550254.open-xchange@omgreatgod.store> Message-ID: Hi Christian, Christian Sell writes: > I have successfully implemented the embedded schema procedure as given > in the XSD examples under Win32. Now I have ported everything over to > Linux-gcc64, and got the compile to succeed (always with newest Xerces > and XSD). However, when I invoke the program, I get a > "xercesc_3_1::OutOfMemoryException". > > When debugging, I see the error happening ate the following statement: > > gp->deserializeGrammars(&is); My first guess is that you are using the schema grammar generated for Win32 to build your 64-bit Linux application. Generally, you should generate the schema grammar on the same platform and using the same Xerces-C++ version as your application. Boris From christian at gsvitec.com Fri Oct 3 04:03:32 2014 From: christian at gsvitec.com (Christian Sell) Date: Fri Oct 3 04:03:35 2014 Subject: [xsd-users] OutOfMemoryException when loading binary schema In-Reply-To: References: <1803753366.1714497.1412276550254.open-xchange@omgreatgod.store> Message-ID: <379064286.1748684.1412323412370.open-xchange@omgreatgod.store> > Boris Kolpackov hat am 3. Oktober 2014 um 08:15 > geschrieben: > > My first guess is that you are using the schema grammar generated for > Win32 to build your 64-bit Linux application. Generally, you should > generate the schema grammar on the same platform and using the same > Xerces-C++ version as your application. your guess was right, Boris. In fact I had re-generated the binary schema (with a newly compiled xsdbin tool), BUT for some reason the output files did not overwrite the original ones and thus weren't used in the compile. Thanks! Christian From svetlana.samsonik at thomsonreuters.com Sun Oct 5 20:22:50 2014 From: svetlana.samsonik at thomsonreuters.com (svetlana.samsonik@thomsonreuters.com) Date: Sun Oct 5 20:23:00 2014 Subject: [xsd-users] Missing include in version 4.0.0 Message-ID: <1BAB72DB2329DF4CA1735854E449C0AD1D5C3F3D@C111TYXEMBX72.ERF.thomson.com> Hello, Generated sources for the schema below (test.xsd) don't compile because of missing include file. Manually correcting generated code to include ItemMetadataType.hxx corrects it. The command used to generate the code: ../bin/xsd cxx-tree --std c++11 \ --root-element newsItem \ --generate-forward \ --file-per-type ../test.xsd Compiled on Linux with gcc 4.4.6. g++ -DHAVE_CONFIG_H -I. -Ilibxsd -Wno-deprecated -DBOOST_DATE_TIME_POSIX_TIME_STD_CONFIG -DTIXML_USE_STL -DBOOST_LOG_DYN_LINK -I./../ucdpxsd -I./../ucdpxsd/libxsd -I./../ucdpxsd/cxx-classes -I./../libs/boost -I./../libs/xerces/include -pthread -ldl -lrt -ggdb3 -std=c++0x -O0 -DDEBUG -fPIC -rdynamic -MT cxx-classes/newsItem.lo -MD -MP -MF cxx-classes/.deps/newsItem.Tpo -c cxx-classes/newsItem.cxx -fPIC -DPIC -o cxx-classes/.libs/newsItem.o Regards, Svetlana From christian at gsvitec.com Mon Oct 6 09:35:57 2014 From: christian at gsvitec.com (Christian Sell) Date: Mon Oct 6 09:36:01 2014 Subject: [xsd-users] XSD or XSD/e Message-ID: <001b01cfe16a$760dc2b0$62294810$@gsvitec.com> Hello, we are writing an application that needs XML data binding on multiple platforms, ranging from Windows desktop to Android and iOS tablets. I am uncertain which product to use. Thanks, Christian Sell From boris at codesynthesis.com Mon Oct 6 09:54:39 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Oct 6 10:01:49 2014 Subject: [xsd-users] Missing include in version 4.0.0 In-Reply-To: <1BAB72DB2329DF4CA1735854E449C0AD1D5C3F3D@C111TYXEMBX72.ERF.thomson.com> References: <1BAB72DB2329DF4CA1735854E449C0AD1D5C3F3D@C111TYXEMBX72.ERF.thomson.com> Message-ID: Svetlana, Thanks for the bug report and the test case. I believe we have already fixed this: http://scm.codesynthesis.com/?p=xsd/xsd.git;a=commit;h=169b95d1c079bcc12c2a9bc881ecf2cccf0b6029 If you would like, I can build you a pre-release binary for Linux with this fix. Boris From boris at codesynthesis.com Mon Oct 6 09:57:00 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Oct 6 10:04:11 2014 Subject: [xsd-users] XSD or XSD/e In-Reply-To: <001b01cfe16a$760dc2b0$62294810$@gsvitec.com> References: <001b01cfe16a$760dc2b0$62294810$@gsvitec.com> Message-ID: Hi Christian, Christian Sell writes: > we are writing an application that needs XML data binding on multiple > platforms, ranging from Windows desktop to Android and iOS tablets. I am > uncertain which product to use. For Andorid and iOS, I would go with XSD/e. Boris From christian at gsvitec.com Mon Oct 6 10:06:56 2014 From: christian at gsvitec.com (Christian Sell) Date: Mon Oct 6 10:07:00 2014 Subject: AW: [xsd-users] XSD or XSD/e In-Reply-To: References: <001b01cfe16a$760dc2b0$62294810$@gsvitec.com> Message-ID: <002801cfe16e$c9c53330$5d4f9990$@gsvitec.com> Hello Boris, could you elaborate a little more, please? Could we use one tool for all? Would code that is written against the XSD-generated classes also compile against XSD/e generated classes? Thanks, Christian -----Urspr?ngliche Nachricht----- Von: Boris Kolpackov [mailto:boris@codesynthesis.com] Gesendet: Montag, 6. Oktober 2014 15:57 An: Christian Sell Cc: xsd-users@codesynthesis.com Betreff: Re: [xsd-users] XSD or XSD/e Hi Christian, Christian Sell writes: > we are writing an application that needs XML data binding on multiple > platforms, ranging from Windows desktop to Android and iOS tablets. I > am uncertain which product to use. For Andorid and iOS, I would go with XSD/e. Boris From boris at codesynthesis.com Mon Oct 6 10:02:45 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Oct 6 10:09:56 2014 Subject: [xsd-users] XSD or XSD/e In-Reply-To: <002801cfe16e$c9c53330$5d4f9990$@gsvitec.com> References: <001b01cfe16a$760dc2b0$62294810$@gsvitec.com> <002801cfe16e$c9c53330$5d4f9990$@gsvitec.com> Message-ID: Hi Christian, Christian Sell writes: > could you elaborate a little more, please? Could we use one tool for all? Yes, sure. Plenty of people use XSD/e for desktop/server development, mainly because it is dependency-free. > Would code that is written against the XSD-generated classes also compile > against XSD/e generated classes? No, you definitely don't want to go multi-tool route unless you have a really good reason. It is much simpler to just use XSD/e everywhere. Boris From christian at gsvitec.com Mon Oct 6 10:15:09 2014 From: christian at gsvitec.com (Christian Sell) Date: Mon Oct 6 10:15:13 2014 Subject: AW: [xsd-users] XSD or XSD/e In-Reply-To: References: <001b01cfe16a$760dc2b0$62294810$@gsvitec.com> <002801cfe16e$c9c53330$5d4f9990$@gsvitec.com> Message-ID: <002a01cfe16f$effc69f0$cff53dd0$@gsvitec.com> Ok, thanks. Now could you provide a reason why ever to use XSD and not XSD/e? We are using embedded schema - does that work with XSD/e (I assume it's automatic)? How about commercial pricing? Do you think Xerces is so much overhead that it precludes deployment to Android/iOS devices (with > 1 Gig RAM nowadays)? Thanks C. -----Urspr?ngliche Nachricht----- Von: Boris Kolpackov [mailto:boris@codesynthesis.com] Gesendet: Montag, 6. Oktober 2014 16:03 An: Christian Sell Cc: xsd-users@codesynthesis.com Betreff: Re: [xsd-users] XSD or XSD/e Hi Christian, Christian Sell writes: > could you elaborate a little more, please? Could we use one tool for all? Yes, sure. Plenty of people use XSD/e for desktop/server development, mainly because it is dependency-free. > Would code that is written against the XSD-generated classes also > compile against XSD/e generated classes? No, you definitely don't want to go multi-tool route unless you have a really good reason. It is much simpler to just use XSD/e everywhere. Boris From svetlana.samsonik at thomsonreuters.com Mon Oct 6 11:56:52 2014 From: svetlana.samsonik at thomsonreuters.com (svetlana.samsonik@thomsonreuters.com) Date: Mon Oct 6 11:57:01 2014 Subject: [xsd-users] Missing include in version 4.0.0 In-Reply-To: References: <1BAB72DB2329DF4CA1735854E449C0AD1D5C3F3D@C111TYXEMBX72.ERF.thomson.com> Message-ID: <1BAB72DB2329DF4CA1735854E449C0AD1D5C52C2@C111TYXEMBX72.ERF.thomson.com> Boris, Yes, I would like a pre-release binary. Thank you, Svetlana -----Original Message----- From: Boris Kolpackov [mailto:boris@codesynthesis.com] Sent: Monday, October 06, 2014 9:55 AM To: Samsonik, Svetlana (TR Technology) Cc: xsd-users@codesynthesis.com Subject: Re: [xsd-users] Missing include in version 4.0.0 Svetlana, Thanks for the bug report and the test case. I believe we have already fixed this: http://scm.codesynthesis.com/?p=xsd/xsd.git;a=commit;h=169b95d1c079bcc12c2a9bc881ecf2cccf0b6029 If you would like, I can build you a pre-release binary for Linux with this fix. Boris From laurence.davies at unsw.edu.au Mon Oct 6 19:38:38 2014 From: laurence.davies at unsw.edu.au (Laurence Davies) Date: Mon Oct 6 19:38:48 2014 Subject: [xsd-users] cxx-tree Export Symbols Multiple Constants or Generate Def file via Namespace Map Message-ID: Hi all, Firstly, apologies for the apparent ignorance of my question, I'm very new to both Visual Studio and to XSD (although I am well versed in g++). I have a very very large schema based on GML which I am extending with a smaller subset of types. I am creating a dll for reading/writing to this schema and I would like to have the option of exporting (via _dllspec or otherwise) *only* the subset I have created plus relevant dependencies in GML rather than *all* of GML. This is because the latter is enormous, and even with precompiled headers this vastly increases the complexity of the product I need to deliver. I see that --export-symbol= is the option to use for a blanket all-export or all-import switch, but is there a convenient way to export via namespace? I know this is far more complicated than it sounds due to dependencies but I believe VS10C++ will automatically traverse the inheritance/dependency tree to export all required symbols given that a particular leaf class type is exported. Alternatively, can XSD generate a .def file for this purpose? Regards, Laurence From boris at codesynthesis.com Tue Oct 7 02:59:03 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Oct 7 03:06:14 2014 Subject: [xsd-users] Missing include in version 4.0.0 In-Reply-To: <1BAB72DB2329DF4CA1735854E449C0AD1D5C52C2@C111TYXEMBX72.ERF.thomson.com> References: <1BAB72DB2329DF4CA1735854E449C0AD1D5C3F3D@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5C52C2@C111TYXEMBX72.ERF.thomson.com> Message-ID: Hi Svetlana, svetlana.samsonik@thomsonreuters.com writes: > Yes, I would like a pre-release binary. Here you go: http://codesynthesis.com/~boris/tmp/xsd/xsd-4.1.0.a4-i686-linux-gnu.tar.bz2 Let me know if there are any issues. Boris From boris at codesynthesis.com Tue Oct 7 10:13:26 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Oct 7 10:20:39 2014 Subject: [xsd-users] XSD or XSD/e In-Reply-To: <002a01cfe16f$effc69f0$cff53dd0$@gsvitec.com> References: <001b01cfe16a$760dc2b0$62294810$@gsvitec.com> <002801cfe16e$c9c53330$5d4f9990$@gsvitec.com> <002a01cfe16f$effc69f0$cff53dd0$@gsvitec.com> Message-ID: Hi Christian, Christian Sell writes: > Now could you provide a reason why ever to use XSD and not XSD/e? XSD has more features, is more powerful, flexible, and convenient. However, you pay for that with larger footprint and slightly worse performance, compared to XSD/e. There is also an external dependency on the underlying XML parser library (Xerces-C++). It is best suited for general-purpose development (desktop, server, etc). While XSD/e is not as powerful as XSD, it is smaller, faster and external dependency-free. It was also specifically designed for working in resource-constrained environments, such as mobile and embedded systems. As a result, there are a number of unique features, such as partially in-memory and partially event-driven XML processing, that allow you to handle large documents using a limited amount of memory. > We are using embedded schema - does that work with XSD/e (I assume > it's automatic)? XSD/e implements validation (for both parsing and serialization) of a subset of XML Schema in the generated code. For the list of XML Schema constructs that are validated, refer to Appendix A in the Embedded C++/Parser and C++ Serializer mapping guides. > Do you think Xerces is so much overhead that it precludes deployment > to Android/iOS devices (with > 1 Gig RAM nowadays)? I am not sure about runtime overheads, but the biggest obstacle is actually building Xerces-C++ for Android/iOS. Xerces-C++ is what I would call a "framework" in a sense that it has (or requires) APIs for everything from file IO, to concurrency, to network access. As a result, it has a special layer or code for each operating system (e.g., Win32, POSIX, Mac OS) that abstract these. I don't believe there is anything like this for Android or iOS. But, if you need it badly enough, I am sure Xerces-C++ can be ported to these mobile platforms. I know I've built it some time ago for LynxOS. Boris From svetlana.samsonik at thomsonreuters.com Tue Oct 7 10:24:55 2014 From: svetlana.samsonik at thomsonreuters.com (svetlana.samsonik@thomsonreuters.com) Date: Tue Oct 7 10:25:05 2014 Subject: [xsd-users] Missing include in version 4.0.0 In-Reply-To: References: <1BAB72DB2329DF4CA1735854E449C0AD1D5C3F3D@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5C52C2@C111TYXEMBX72.ERF.thomson.com> Message-ID: <1BAB72DB2329DF4CA1735854E449C0AD1D5C6CA4@C111TYXEMBX72.ERF.thomson.com> Thanks a lot! I will let you know as soon I test it. Svetlana -----Original Message----- From: Boris Kolpackov [mailto:boris@codesynthesis.com] Sent: Tuesday, October 07, 2014 2:59 AM To: Samsonik, Svetlana (TR Technology) Cc: xsd-users@codesynthesis.com Subject: Re: [xsd-users] Missing include in version 4.0.0 Hi Svetlana, svetlana.samsonik@thomsonreuters.com writes: > Yes, I would like a pre-release binary. Here you go: http://codesynthesis.com/~boris/tmp/xsd/xsd-4.1.0.a4-i686-linux-gnu.tar.bz2 Let me know if there are any issues. Boris From boris at codesynthesis.com Wed Oct 8 01:53:48 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Oct 8 02:00:59 2014 Subject: [xsd-users] cxx-tree Export Symbols Multiple Constants or Generate Def file via Namespace Map In-Reply-To: References: Message-ID: Hi Laurence, Laurence Davies writes: > I have a very very large schema based on GML which I am extending with a > smaller subset of types. I am creating a dll for reading/writing to this > schema and I would like to have the option of exporting (via _dllspec or > otherwise) *only* the subset I have created plus relevant dependencies in > GML rather than *all* of GML. This is because the latter is enormous, and > even with precompiled headers this vastly increases the complexity of the > product I need to deliver. > > I see that --export-symbol= is the option to use for a blanket all-export > or all-import switch, but is there a convenient way to export via > namespace? I know this is far more complicated than it sounds due > to dependencies but I believe VS10C++ will automatically traverse > the inheritance/dependency tree to export all required symbols given > that a particular leaf class type is exported. You are correct about auto-exporting of base classes (which, BTW, is a major source of trouble). There is, however, no auto-exporting of classes that are merely "used", e.g., as data members or in the API. So in this case you will have to somehow manually track down all the used classes and export them, transitively. In case of GML, this will be a monumental amount of work. Also, keep in mind that GML is a polymorphic schema which means certain classes might not be "used" (in the above sense) but still might have to be exported since they will be created and returned at runtime. In the end, once you take all this into account, I am pretty sure you will end up with exporting pretty much all of the GML. You can also look at this problem in another way: if you are not exporting certain classes, then why have them in the DLL in the first place? So what you may want to consider is simply removing the parts of the GML that you don't need. In fact, for GML 3.1.1 there was the "small profile" (3.1.1s). You may want to check if there is something like this for the version that you are using. > Alternatively, can XSD generate a .def file for this purpose? No, XSD doesn't have this functionality. Boris From laurence.davies at unsw.edu.au Wed Oct 8 22:32:59 2014 From: laurence.davies at unsw.edu.au (Laurence Davies) Date: Wed Oct 8 22:33:23 2014 Subject: [xsd-users] cxx-tree Export Symbols Multiple Constants or Generate Def file via Namespace Map In-Reply-To: References: , Message-ID: Hi Boris, Thank you for your valuable suggestions. > simply removing the parts of the GML that you don't need. This is in itself a very large task! > In fact, for GML 3.1.1 there was the "small profile" (3.1.1s). You may want to check if there is something like this for the version that you are using. Do you mean the Simple Features profile? There are also the Point Profile and the CRSSupport Profiles. Unfortunately, our schema uses some of the types not included in these profiles, specifically gml:AbstractCRS and some others, thus ruling out an off-the-shelf solution. I noticed that there appears to be mention of a XSLT subset tool for GML in the specification and also on various websites, but it appears to be missing from any repository. Would you know anything about this tool? Laurence Davies ____________________ Research Assistant in eGeodesy CRC-SI, UNSW ________________________________________ From: Boris Kolpackov [boris@codesynthesis.com] Sent: Wednesday, 8 October 2014 4:53 PM To: Laurence Davies Cc: xsd-users@codesynthesis.com Subject: Re: [xsd-users] cxx-tree Export Symbols Multiple Constants or Generate Def file via Namespace Map Hi Laurence, Laurence Davies writes: > I have a very very large schema based on GML which I am extending with a > smaller subset of types. I am creating a dll for reading/writing to this > schema and I would like to have the option of exporting (via _dllspec or > otherwise) *only* the subset I have created plus relevant dependencies in > GML rather than *all* of GML. This is because the latter is enormous, and > even with precompiled headers this vastly increases the complexity of the > product I need to deliver. > > I see that --export-symbol= is the option to use for a blanket all-export > or all-import switch, but is there a convenient way to export via > namespace? I know this is far more complicated than it sounds due > to dependencies but I believe VS10C++ will automatically traverse > the inheritance/dependency tree to export all required symbols given > that a particular leaf class type is exported. You are correct about auto-exporting of base classes (which, BTW, is a major source of trouble). There is, however, no auto-exporting of classes that are merely "used", e.g., as data members or in the API. So in this case you will have to somehow manually track down all the used classes and export them, transitively. In case of GML, this will be a monumental amount of work. Also, keep in mind that GML is a polymorphic schema which means certain classes might not be "used" (in the above sense) but still might have to be exported since they will be created and returned at runtime. In the end, once you take all this into account, I am pretty sure you will end up with exporting pretty much all of the GML. You can also look at this problem in another way: if you are not exporting certain classes, then why have them in the DLL in the first place? So what you may want to consider is simply removing the parts of the GML that you don't need. In fact, for GML 3.1.1 there was the "small profile" (3.1.1s). You may want to check if there is something like this for the version that you are using. > Alternatively, can XSD generate a .def file for this purpose? No, XSD doesn't have this functionality. Boris From boris at codesynthesis.com Thu Oct 9 00:33:37 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Oct 9 00:40:50 2014 Subject: [xsd-users] cxx-tree Export Symbols Multiple Constants or Generate Def file via Namespace Map In-Reply-To: References: Message-ID: Hi Laurence, Laurence Davies writes: > > simply removing the parts of the GML that you don't need. > > This is in itself a very large task! Trying to export only a subset of GML symbols is even larger. > > In fact, for GML 3.1.1 there was the "small profile" (3.1.1s). You > > may want to check if there is something like this for the version that > > you are using. > > Do you mean the Simple Features profile? I am not 100% sure what exactly it is called. There was a schema called 3.1.1s in the GML repository that was significantly smaller than 3.1.1. The README says that it has all the deprecated stuff removed. > I noticed that there appears to be mention of a XSLT subset tool for > GML in the specification and also on various websites, but it appears > to be missing from any repository. Would you know anything about this > tool? No, I have never heard of it. Perhaps it makes sense to ask the GML folks? Generally, I am not a GML expert. In fact, what I know about GML is already way more than what I wish I had to know ;-). Boris From christian at gsvitec.com Fri Oct 10 04:07:34 2014 From: christian at gsvitec.com (Christian Sell) Date: Fri Oct 10 04:07:37 2014 Subject: [xsd-users] handling polymorphism Message-ID: <296760820.146445.1412928454280.open-xchange@patina.store> Hello, I am sure this is a standard issue, and the solution is probably described somewhere in the docs. Nonetheless I am trying the shortcut through the mailing list, maybe someone can provide a pointer to the relevant chapters: Our schema contains a type hierarchy, which is also represented as a substitution group. At the root of the hierarchy (and the sgroup) lies "abstractitem". abstractitem is also referenced elswhere as contents of an unbounded sequence. Now, when processing the sequence, we need to apply specific algorithms to individual types. Ideally, we would like to have a true "object-oriented" solution, like: //declared in abstractitem: virtual void process(Processor proc) = 0; //implemented in subtypes: void process(Processor proc) { proc.processSpecificType(this); } can this be achieved with XSD(/e)? Or is there another solution? Please answer with special focus on XSD/e, as we will have to migrate there thanks, Christian Sell From vladimir.zykov at ncloudtech.ru Fri Oct 10 06:00:46 2014 From: vladimir.zykov at ncloudtech.ru (Vladimir Zykov) Date: Fri Oct 10 09:11:17 2014 Subject: [xsd-users] Ignoring unknown elements Message-ID: Hi Boris, In our problem domain we cannot be sure about validity (in XML sense) of input XML documents. I.e. input document can contain elements/attributes not present in XML Schema. Order of elements is preserved as per XML Schema though. While it's easy to ignore unknown attributes (just providing dont_validate flag) ignoring unknown elements requires changes in XSD code generation. Currently generated code stops processing of XML nodes in complex type when unknown element is encountered. We generate code in cxx-tree mode. I found the following discussion about this problem: http://www.codesynthesis.com/pipermail/xsd-users/2008-November/002046.html It was back in 2008. Does the problem still exist? Looking at the XSD command line arguments I don't seem to see anything that could fix this problem. I understand your explanation from the email above, however, if I know that my XML Schema files don't use any kind of type derivation then bailing out on any unknown element might be too restrictive for me. Is there anything else that hinders ignoring unknown elements? Shallow inspection of generated code shows that the problem can be fixed by mere removal of 'break' statement at the end of a for-loop that processes child elements in generated parse() function. Certainly this should be managed by command line option. I think it can be useful for your users. What do you think? Thanks, Vladimir Zykov Software Engineer New Cloud Technologies, LLC From boris at codesynthesis.com Mon Oct 13 01:10:04 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Oct 13 01:17:19 2014 Subject: [xsd-users] handling polymorphism In-Reply-To: <296760820.146445.1412928454280.open-xchange@patina.store> References: <296760820.146445.1412928454280.open-xchange@patina.store> Message-ID: Hi Christian, Christian Sell writes: > can this be achieved with XSD(/e)? Yes, for XSD, see the C++/Tree Mapping Customization Guide: http://wiki.codesynthesis.com/Tree/Customization_guide As well as the examples in the examples/cxx/tree/custom/ directory, specifically, 'taxonomy'. For XSD/e, see Section 4.9, "Customizing the Object Model" in the C++/Hybrid Getting Started Guide. > Please answer with special focus on XSD/e, as we will have to > migrate there This is a wrong mailing list for asking questions about XSD/e. It has its own mailing list: xsde-users@codesynthesis.com Boris From boris at codesynthesis.com Tue Oct 14 01:38:08 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Oct 14 01:45:22 2014 Subject: [xsd-users] Ignoring unknown elements In-Reply-To: References: Message-ID: Hi Vladimir, Vladimir Zykov writes: > In our problem domain we cannot be sure about validity (in XML sense) of > input XML documents. > > [...] > > Shallow inspection of generated code shows that the problem can be fixed > by mere removal of 'break' statement at the end of a for-loop that > processes child elements in generated parse() function. Certainly this > should be managed by command line option. I am not certain about this. You are trying to parse an invalid XML instance and this is quite unusual. While the "fix" seems simple, the problem is that it will break (at runtime) if someone derives from such a type. So the XSD compiler would probably need to check that this doesn't happen. But that is not always possible (e.g., if the derived type is in another schema). So the "fix" becomes a lot less trivial and robust. So before we go deeper into this, isn't there a cleaner way to achieve the same end result. The two options that I see: 1. Add wildcards to your XML Schema types. I remember you told me that you auto-generate the schema so maybe this can be done fairly easily? 2. If the "extra" elements have a regular form (e.g., they are in a different namespace), then what you could do is pre-process the DOM representation of your document before handing it off to the XSD-generated code. The idea is to remove all the extra elements during this step. Boris From torokze at gmail.com Tue Oct 14 01:58:04 2014 From: torokze at gmail.com (=?UTF-8?B?Wm9sdMOhbiBUw7Zyw7Zr?=) Date: Tue Oct 14 01:58:11 2014 Subject: [xsd-users] 'DOMDocument' : ambiguous symbol Message-ID: Hi, XSD version 4.0.0 Xerces version 3.1.1 xsd-4.0.0-i686-windows\libxsd\xsd\cxx\xml\dom\parsing-source.txx(363): error C2872: 'DOMDocument' : ambiguous symbol ? ? could be 'c:\program files (x86)\microsoft sdks\windows\v7.0a\include\msxml.h(171) : DOMDocument' ? ? or 'c:\projects\opsys\opc\thirdparty\xerces-c-3.1.1-x86-windows-vc-10.0\include\xercesc\dom\domdocument.hpp(64) : xercesc_3_1::DOMDocument' ?There are two places in parsing-source.txx where the ?DOMDocument has no namespace prefix like xercesc::DOMDocument. Regards, Zolee From boris at codesynthesis.com Tue Oct 14 09:38:15 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Oct 14 09:45:29 2014 Subject: [xsd-users] 'DOMDocument' : ambiguous symbol In-Reply-To: References: Message-ID: Hi Zolt?n, Thanks for the report. This has already been fixed for the upcoming release. You can also try a pre-release binary: http://codesynthesis.com/~boris/tmp/xsd/xsd-4.1.0.a3-i686-windows.zip Boris From info at rriskml.com Tue Oct 14 12:41:29 2014 From: info at rriskml.com (Vall Herard) Date: Wed Oct 15 00:51:38 2014 Subject: [xsd-users] XSD, Boost archive and EXI (xml compression) compatibility Message-ID: <1413304887736.91366@rriskml.com> We need a small piece of software using the Boost archive interface so it can be easily integrated into XSD to help with xml EXI implementation. We are searching for a consultant who can integrate EXI (compressed XML standard from AgileDelta) and the CodeSynthesis XSD translator so that the latter can parse and serialize schema-compliant EXI documents. In other words, we need software to mediate between XSD C++ data objects and EXI compressed XML directly rather than requiring decompression. The ideal person to work on this would have experience in all three areas: (i) CodeSynthesis XSD translator, (ii) Boost archive library, and (iii) the EFX implementation of EXI provided by AgileDelta, although experience with other EXI implementations would also be acceptable.? Can any one help? If you are interested and posses the needed skills, place contact Michael Rose at michaerose@msn.com or info@rriskml.com From svetlana.samsonik at thomsonreuters.com Wed Oct 15 11:36:07 2014 From: svetlana.samsonik at thomsonreuters.com (svetlana.samsonik@thomsonreuters.com) Date: Wed Oct 15 11:36:17 2014 Subject: [xsd-users] Missing include in version 4.0.0 In-Reply-To: References: <1BAB72DB2329DF4CA1735854E449C0AD1D5C3F3D@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5C52C2@C111TYXEMBX72.ERF.thomson.com> Message-ID: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3AFB@C111TYXEMBX72.ERF.thomson.com> Hi Boris, No issues found with the pre-release library. Thank you, Svetlana -----Original Message----- From: Boris Kolpackov [mailto:boris@codesynthesis.com] Sent: Tuesday, October 07, 2014 2:59 AM To: Samsonik, Svetlana (TR Technology) Cc: xsd-users@codesynthesis.com Subject: Re: [xsd-users] Missing include in version 4.0.0 Hi Svetlana, svetlana.samsonik@thomsonreuters.com writes: > Yes, I would like a pre-release binary. Here you go: http://codesynthesis.com/~boris/tmp/xsd/xsd-4.1.0.a4-i686-linux-gnu.tar.bz2 Let me know if there are any issues. Boris From svetlana.samsonik at thomsonreuters.com Wed Oct 15 18:16:13 2014 From: svetlana.samsonik at thomsonreuters.com (svetlana.samsonik@thomsonreuters.com) Date: Wed Oct 15 18:16:23 2014 Subject: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 Message-ID: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3F49@C111TYXEMBX72.ERF.thomson.com> Hi Boris, After upgrading to version 4.0 there is a performance difference for parsing and binary serialization (XDR). Here are my metrics in microseconds for the same file processed: version 3.3.0 version 4.0.0 Parse XML to C++/Tree object: 4644 6845 Serialize C++/Tree object to XRD: 227 303 Copy from XDR to C++/Tree object: 445 855 Serialize C++/Tree object to XML: 3550 4140 Is the difference expected? Is there a way to improve the performance for version 4.0? Thank you, Svetlana From boris at codesynthesis.com Wed Oct 15 23:32:00 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Oct 15 23:39:15 2014 Subject: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 In-Reply-To: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3F49@C111TYXEMBX72.ERF.thomson.com> References: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3F49@C111TYXEMBX72.ERF.thomson.com> Message-ID: Hi Svetlana, svetlana.samsonik@thomsonreuters.com writes: > After upgrading to version 4.0 there is a performance difference for > parsing and binary serialization (XDR). > > [...] > > Is the difference expected? We check for parsing/serialization performance regressions before every release and there weren't any between 3.3.0 and 4.0.0. We haven't tested binary serialization but I wouldn't expect any difference there since it stayed relatively unchanged. There is the 'performance' example that measures the parsing and serialization performance. Here are the numbers I get for a 50k file: ./driver-3.3 test-50k.xml parsing: document size: 51324 bytes iterations: 1000 time: 3.628948000 sec throughput: 275.562 documents/sec throughput: 13.4878 MBytes/sec serialization: document size: 51324 bytes iterations: 1000 time: 4.977265000 sec throughput: 200.914 documents/sec throughput: 9.83399 MBytes/sec ./driver-4.0 test-50k.xml parsing: document size: 51324 bytes iterations: 1000 time: 3.686281000 sec throughput: 271.276 documents/sec throughput: 13.278 MBytes/sec serialization: document size: 51324 bytes iterations: 1000 time: 5.215050000 sec throughput: 191.753 documents/sec throughput: 9.3856 MBytes/sec For 500k file: ./driver-3.3 test-500k.xml parsing: document size: 512146 bytes iterations: 1000 time: 38.327591000 sec throughput: 26.0909 documents/sec throughput: 12.7433 MBytes/sec serialization: document size: 512146 bytes iterations: 1000 time: 51.091402000 sec throughput: 19.5728 documents/sec throughput: 9.55974 MBytes/sec ./driver-4.0 test-500k.xml parsing: document size: 512146 bytes iterations: 1000 time: 39.594913000 sec throughput: 25.2558 documents/sec throughput: 12.3354 MBytes/sec serialization: document size: 512146 bytes iterations: 1000 time: 50.190411000 sec throughput: 19.9241 documents/sec throughput: 9.73135 MBytes/sec This is on my Linux box compiled with GCC/-O3 and using Xerces-C++ 3.1.1. Could you run this test in your configuration and see if you get a similar picture? Boris From vladimir.zykov at ncloudtech.ru Thu Oct 16 05:58:16 2014 From: vladimir.zykov at ncloudtech.ru (Vladimir Zykov) Date: Thu Oct 16 06:49:24 2014 Subject: [xsd-users] Ignoring unknown elements In-Reply-To: References: Message-ID: Hi Boris, I see now that I must clarify a bit what I described in my previous email. And it seems I already made attempts to do so. http://codesynthesis.com/pipermail/xsd-users/2014-May/004292.html So, from the discussion above I disabled XML validation and it worked fine for XML attributes. Obviously, it didn't work for XML elements as we discuss it now. Now from the very beginning. We have 2 sets of schema files: one for Office Open XML and one for ODF. While specification for Office Open XML is bundled with its XML schema files ODF provides only RelaxNG schema and we had to covert it to XML Schema. Both of these schemas are very big and consequently XSD generates a lot of code from them. Back to the problem. Both these specifications allow extensions to the file format and this is true that those extensions elements will be in different namespace. However, preprocessing XML DOM will require adding that preprocessing code (implying higher maintenance cost) and will require more processing in runtime. :( Adding wildcards will certainly work but this will also add more generated code and probably higher memory consumption for things that we don't need. What we need instead is processing of elements that we know (i.e. we copy data from those elements to our representation) and ignore everything else. There is another related problem. When we compile generated code in debug mode we obtain 2 libraries (OOXML and ODF) of comparable size. It's about 2.5GB each which results in longer compilation time and dramatical linking time. On MacOS we crash linker (it asserts on some internal data which exceeds int32 value) and on Linux linker requires so much memory that the only way to link executable is to do it in 1 job (compilation in, say, 8 jobs is fine). So, you see. We would rather remove some code instead of adding more. :) In order to achieve this we are ready to rewrite our XSD schema files (we do this anyway for ODF) and remove from them things that we don't currently support so that what was in the specification will become unknown content to the generated code. We also remove any derivation (by extension or by restriction) for all complex types with complex content (i.e. we copy content from base type to the derived type). We are fine with this and you see that the change in XSD that we originally proposed will work for us and will not break anything. Note to the possible implementation. The mode in which generated code will ignore unknown elements must be turned on with XSD command line option and certainly this mode is not compatible with XML Schema files in which complex types with complex content use derivation mechanisms. Code generation in this case should fail, I think. Thanks, Vladimir Zykov Software Engineer New Cloud Technologies, LLC On Oct 14, 2014, at 8:38 AM, Boris Kolpackov wrote: > Hi Vladimir, > > Vladimir Zykov writes: > >> In our problem domain we cannot be sure about validity (in XML sense) of >> input XML documents. >> >> [...] >> >> Shallow inspection of generated code shows that the problem can be fixed >> by mere removal of 'break' statement at the end of a for-loop that >> processes child elements in generated parse() function. Certainly this >> should be managed by command line option. > > I am not certain about this. You are trying to parse an invalid XML > instance and this is quite unusual. > > While the "fix" seems simple, the problem is that it will break (at > runtime) if someone derives from such a type. So the XSD compiler > would probably need to check that this doesn't happen. But that is > not always possible (e.g., if the derived type is in another schema). > So the "fix" becomes a lot less trivial and robust. > > So before we go deeper into this, isn't there a cleaner way to > achieve the same end result. The two options that I see: > > 1. Add wildcards to your XML Schema types. I remember you told me that > you auto-generate the schema so maybe this can be done fairly easily? > > 2. If the "extra" elements have a regular form (e.g., they are in a > different namespace), then what you could do is pre-process the > DOM representation of your document before handing it off to the > XSD-generated code. The idea is to remove all the extra elements > during this step. > > Boris From boris at codesynthesis.com Thu Oct 16 07:02:41 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Oct 16 07:09:56 2014 Subject: [xsd-users] Ignoring unknown elements In-Reply-To: References: Message-ID: Hi Vladimir, Vladimir Zykov writes: > Back to the problem. Both these specifications allow extensions to the file > format and this is true that those extensions elements will be in different > namespace. However, preprocessing XML DOM will require adding that preprocessing > code (implying higher maintenance cost) I realize having us maintain a feature that's made especially for you (and that nobody else will use) is easier but we don't consider this good engineering. Plus, the code that removes all elements from a DOM document that belong to a certain namespace will probably take half a page. > and will require more processing in runtime. It is runtime processing one way or the other. > In order to achieve this we are ready to rewrite our XSD schema files (we do this > anyway for ODF) and remove from them things that we don't currently support so > that what was in the specification will become unknown content to the generated > code. > We also remove any derivation (by extension or by restriction) for all > complex types with complex content (i.e. we copy content from base type > to the derived type). This could very well be the reason why you end up with huge libraries. You are essentially duplicating the same code over and over again instead of simply reusing the base classes. > Note to the possible implementation. The mode in which generated code > will ignore unknown elements must be turned on with XSD command line > option and certainly this mode is not compatible with XML Schema files > in which complex types with complex content use derivation mechanisms. Which is almost any real-world schema. > Code generation in this case should fail, I think. Yes, this will work for your very unique case. What about someone else who also has derivation? They could reasonably expect to be able to specify individual types that should have content ignored. The point I am trying to make is that we don't add features that are only usable by a single user. Boris From vladimir.zykov at ncloudtech.ru Thu Oct 16 07:49:08 2014 From: vladimir.zykov at ncloudtech.ru (Vladimir Zykov) Date: Thu Oct 16 08:54:36 2014 Subject: [xsd-users] Ignoring unknown elements In-Reply-To: References: Message-ID: Boris, I see your point. See my comments inline. >> In order to achieve this we are ready to rewrite our XSD schema files (we do this >> anyway for ODF) and remove from them things that we don't currently support so >> that what was in the specification will become unknown content to the generated >> code. > >> We also remove any derivation (by extension or by restriction) for all >> complex types with complex content (i.e. we copy content from base type >> to the derived type). > > This could very well be the reason why you end up with huge libraries. > You are essentially duplicating the same code over and over again > instead of simply reusing the base classes. It could be but we don't use it yet, we just considering such a change. For now we use original schemas with type derivations. What I wanted to state is that amount of generated/compiled code is big because schemas are very big. Why we think changing schema will reduce amount of code is because we support a small portion of the original schema. This might change someday but for now we better remove non-supported types/elements. >> Code generation in this case should fail, I think. > > Yes, this will work for your very unique case. What about someone else > who also has derivation? They could reasonably expect to be able to > specify individual types that should have content ignored. The point > I am trying to make is that we don't add features that are only usable > by a single user. That's fine. If you think that very few users will need it we can do this ourselves. The only thing that is unclear for me now is how to publish our changes as demanded by GNU license (XSD is only project with GNU license that we use). But this is a minor problem. Anyway, thanks for taking a look. Vladimir Zykov Software Engineer New Cloud Technologies, LLC From boris at codesynthesis.com Fri Oct 17 08:47:00 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Oct 17 08:54:15 2014 Subject: [xsd-users] Ignoring unknown elements In-Reply-To: References: Message-ID: Hi Vladimir, Vladimir Zykov writes: > Why we think changing schema will reduce amount of code is because we support > a small portion of the original schema. This might change someday but for now > we better remove non-supported types/elements. So you are essentially cleaning up the schema of all the elements that you don't use even though the actual XML might contain them. One thing that you may consider doing is generate a list of "known" elements and then filtering the DOM from all the unknown ones. One way to do this would be to delete them from the DOM document after it has been loaded (and before handing it off to XSD-generated code). But if you are after efficiency, then a better way would be to filter this out while parsing. I vaguely remember mentioning of filters in the DOM API but I myself never used them. The other option is to do something similar to what the 'streaming' example does. With this mechanism you could easily ignore whole XML subtrees (based on your known element list) without them ever ending up in DOM. I can't think of a more efficient way than that. The only potential problem with the known element list approach is that the same element name could be "known" in one context (e.g., XML Schema type) and "unknown" in another. I would resolve this by saying that if an element name is known in one place, then it must be also known in all other places in the schema. > That's fine. If you think that very few users will need it we can do this > ourselves. The only thing that is unclear for me now is how to publish our > changes as demanded by GNU license (XSD is only project with GNU license > that we use). Seeing that you have a proprietary license for XSD, you don't need to worry about the GPL requirements. If you don't want to publish your changes, you don't have to. Boris From svetlana.samsonik at thomsonreuters.com Mon Oct 20 17:59:25 2014 From: svetlana.samsonik at thomsonreuters.com (svetlana.samsonik@thomsonreuters.com) Date: Tue Oct 21 00:38:04 2014 Subject: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 In-Reply-To: References: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3F49@C111TYXEMBX72.ERF.thomson.com> Message-ID: <1BAB72DB2329DF4CA1735854E449C0AD1D5D9AC8@C111TYXEMBX72.ERF.thomson.com> Hi Boris, I ran the tests in the 'performance' example and received very similar results. I my own test are slightly different. I changed the example code to mimic the my logic and I received similar results to yours for parsing and serialization. When I run the those tests for "real" schema (which is complicated) and file size 25K, I receive the following results version 3.3.0 version 4.0.0 (build with --std c++11) microseconds microseconds Parse XML to C++/Tree object: 4719 5420 Serialize C++/Tree object to XML: 5303 5735 Serialize C++/Tree object to XRD: 299 329 Copy from XDR to C++/Tree object: 639 1405 Binary representation size: 32688 30108 Overall the results do show some time increase for version 4.0 for parsing/serialization and binary serialization to XDR. My biggest concern is "Copy from XDR to C++/Tree object" where it is 4 times longer to compare with "Serialize C++/Tree object to XRD" in version 4.0, where it is only 2 times in version 3.3.0. I ran valgrind --tool=callgrind for both test and you can see that the same function (loadFromXDR) has much more "Ir per call" for version 4.0 (first picture) vs version 3.3 ( second picture). Do you know what can cause such behavior? Thank you, Svetlana [cid:part1.00030600.01040202@comcast.net] [cid:part2.08030903.08030108@comcast.net] -----Original Message----- From: Boris Kolpackov [mailto:boris@codesynthesis.com] Sent: Wednesday, October 15, 2014 11:32 PM To: Samsonik, Svetlana (TR Technology) Cc: xsd-users@codesynthesis.com Subject: Re: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 Hi Svetlana, svetlana.samsonik@thomsonreuters.com > writes: > After upgrading to version 4.0 there is a performance difference for > parsing and binary serialization (XDR). > > [...] > > Is the difference expected? We check for parsing/serialization performance regressions before every release and there weren't any between 3.3.0 and 4.0.0. We haven't tested binary serialization but I wouldn't expect any difference there since it stayed relatively unchanged. There is the 'performance' example that measures the parsing and serialization performance. Here are the numbers I get for a 50k file: ./driver-3.3 test-50k.xml parsing: document size: 51324 bytes iterations: 1000 time: 3.628948000 sec throughput: 275.562 documents/sec throughput: 13.4878 MBytes/sec serialization: document size: 51324 bytes iterations: 1000 time: 4.977265000 sec throughput: 200.914 documents/sec throughput: 9.83399 MBytes/sec ./driver-4.0 test-50k.xml parsing: document size: 51324 bytes iterations: 1000 time: 3.686281000 sec throughput: 271.276 documents/sec throughput: 13.278 MBytes/sec serialization: document size: 51324 bytes iterations: 1000 time: 5.215050000 sec throughput: 191.753 documents/sec throughput: 9.3856 MBytes/sec For 500k file: ./driver-3.3 test-500k.xml parsing: document size: 512146 bytes iterations: 1000 time: 38.327591000 sec throughput: 26.0909 documents/sec throughput: 12.7433 MBytes/sec serialization: document size: 512146 bytes iterations: 1000 time: 51.091402000 sec throughput: 19.5728 documents/sec throughput: 9.55974 MBytes/sec ./driver-4.0 test-500k.xml parsing: document size: 512146 bytes iterations: 1000 time: 39.594913000 sec throughput: 25.2558 documents/sec throughput: 12.3354 MBytes/sec serialization: document size: 512146 bytes iterations: 1000 time: 50.190411000 sec throughput: 19.9241 documents/sec throughput: 9.73135 MBytes/sec This is on my Linux box compiled with GCC/-O3 and using Xerces-C++ 3.1.1. Could you run this test in your configuration and see if you get a similar picture? Boris -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 602723 bytes Desc: image001.png Url : http://codesynthesis.com/pipermail/xsd-users/attachments/20141020/5531bee9/image001-0001.png -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 587989 bytes Desc: image002.png Url : http://codesynthesis.com/pipermail/xsd-users/attachments/20141020/5531bee9/image002-0001.png From boris at codesynthesis.com Wed Oct 22 01:35:33 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Oct 22 01:42:45 2014 Subject: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 In-Reply-To: <1BAB72DB2329DF4CA1735854E449C0AD1D5D9AC8@C111TYXEMBX72.ERF.thomson.com> References: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3F49@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5D9AC8@C111TYXEMBX72.ERF.thomson.com> Message-ID: Hi Svetlana, svetlana.samsonik@thomsonreuters.com writes: > I ran the tests in the 'performance' example and received very similar > results. > > I my own test are slightly different. I changed the example code to > mimic the my logic and I received similar results to yours for parsing > and serialization. So, just to confirm, you have changes the 'performance' test to mimic your logic but still used the "dummy" schema that comes with the test? And the result was that the numbers for 3.3.0 and 4.0.0 were very close, correct? > When I run the those tests for "real" schema (which is complicated) > and file size 25K, I receive the following results I am not sure what you mean by "those tests". The output that you showed doesn't look like the 'performance' test. And it doesn't have the binary serialization part. Did you add this to the 'performance'. > My biggest concern is "Copy from XDR to C++/Tree object" where it is > 4 times longer to compare with "Serialize C++/Tree object to XRD" in > version 4.0, where it is only 2 times in version 3.3.0. > > I ran valgrind --tool=callgrind for both test and you can see that > the same function (loadFromXDR) has much more "Ir per call" for > version 4.0 (first picture) vs version 3.3 ( second picture). This is interesting. Could you do two things for me: 1. Build your 4.0.0 test in the c++98 rather than c++11 mode and see if there are any differences. 2. Send the power::newMessage(XDR) constructor body for both 3.3.0 and 4.0.0. I have looked at a few such constructors from the examples and they look almost identical. There are a few extra inline function calls in 4.0.0 but they definitely cannot explain this difference (unless you built with optimization disabled). Boris From svetlana.samsonik at thomsonreuters.com Sun Oct 26 17:52:04 2014 From: svetlana.samsonik at thomsonreuters.com (svetlana.samsonik@thomsonreuters.com) Date: Sun Oct 26 17:52:13 2014 Subject: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 In-Reply-To: References: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3F49@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5D9AC8@C111TYXEMBX72.ERF.thomson.com> Message-ID: <1BAB72DB2329DF4CA1735854E449C0AD1D5EC893@C111TYXEMBX72.ERF.thomson.com> Hi Boris, Please see my answers in blue below: > I ran the tests in the 'performance' example and received very similar > results. > > I my own test are slightly different. I changed the example code to > mimic the my logic and I received similar results to yours for parsing > and serialization. >>So, just to confirm, you have changes the 'performance' test to mimic your logic but still used the "dummy" schema that >>comes with the test? >>And the result was that the numbers for 3.3.0 and 4.0.0 were very close, correct? Yes, correct. > When I run the those tests for "real" schema (which is complicated) > and file size 25K, I receive the following results >>I am not sure what you mean by "those tests". The output that you showed doesn't look like the 'performance' test. And it doesn't have the binary serialization part. Did you add this to the 'performance'. My tests are for "real schema". For parsing and serialization they are the same as 'performance' tests plus have binary serialization measurements. No, did not add binary serialization part to the 'performance' test. > My biggest concern is "Copy from XDR to C++/Tree object" where it is > 4 times longer to compare with "Serialize C++/Tree object to XRD" in > version 4.0, where it is only 2 times in version 3.3.0. > > I ran valgrind --tool=callgrind for both test and you can see that the > same function (loadFromXDR) has much more "Ir per call" for version > 4.0 (first picture) vs version 3.3 ( second picture). >>This is interesting. Could you do two things for me: >>1. Build your 4.0.0 test in the c++98 rather than c++11 mode and see if there are any differences. My new results in microseconds for version 4.0.0 --std c++98 -O0 --std c++98 -O2 --std c++11 -O0 --std c++11 -O2 Parse XML to C++/Tree object: 4715 4667 5420 5648 Serialize C++/Tree object to XML: 4998 5082 5735 6038 Serialize C++/Tree object to XRD: 325 356 329 332 Copy from XDR to C++/Tree object: 725 765 1405 1444 >>2. Send the power::newMessage(XDR) constructor body for both 3.3.0 and 4.0.0. I have looked at a few such constructors from the examples and they look almost identical. There are a few extra inline function calls in 4.0.0 but they definitely cannot explain this difference (unless you built with optimization disabled). Both constructors are identical for 3.3.0 and 4.0.0: namespace power { newsMessage:: newsMessage (::xml_schema::istream< XDR >& s, ::xml_schema::flags f, ::xml_schema::container* c) : ::xml_schema::type (s, f, c), header_ (this), itemSet_ (this) { this->parse (s, f); } Thank you, Svetlana From boris at codesynthesis.com Mon Oct 27 09:28:24 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Oct 27 09:35:41 2014 Subject: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 In-Reply-To: <1BAB72DB2329DF4CA1735854E449C0AD1D5EC893@C111TYXEMBX72.ERF.thomson.com> References: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3F49@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5D9AC8@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5EC893@C111TYXEMBX72.ERF.thomson.com> Message-ID: Hi Svetlana, svetlana.samsonik@thomsonreuters.com writes: > --std c++98 -O0 --std c++98 -O2 --std c++11 -O0 --std c++11 -O2 > > Parse XML: 4715 4667 5420 5648 > > Serialize XML: 4998 5082 5735 6038 > > Serialize XRD: 325 356 329 332 > > Pasre XDR: 725 765 1405 1444 Wow, these numbers are hard to believe. Especially the Parse XDR part. What they tell me are two things: 1. C++11 code is twice as slow as C++98. 2. Code built with optimization disabled (-O0) is faster than with optimization enabled (-O2). #2 is especially hard to believe. Can you maybe double check that the test setup is correct? > Both constructors are identical for 3.3.0 and 4.0.0: > > > namespace power > { > newsMessage:: > newsMessage (::xml_schema::istream< XDR >& s, > ::xml_schema::flags f, > ::xml_schema::container* c) > : ::xml_schema::type (s, f, c), > header_ (this), > itemSet_ (this) > { > this->parse (s, f); > } Can you also send the parse() function bodies as well (the ones that take the XDR stream)? Boris From haydar.kantarci at hotmail.com Tue Oct 28 03:39:25 2014 From: haydar.kantarci at hotmail.com (Haydar Ozan KANTARCI) Date: Tue Oct 28 03:39:33 2014 Subject: [xsd-users] XSD xmlns:tns issue Message-ID: Hello,i am having problems defining my tns type from XSD. My xsd header looks like this;however, when i generate the hxx and cxx files, the parameters with "tns" types doesnt seem to work. My launcher code seems like this; char * in = "hello.xml"; using namespace xercesc; XMLPlatformUtils::Initialize (); try { using namespace TavMap; xml_schema::properties props; props.no_namespace_schema_location("hello.xsd"); auto_ptr plPtr ( pageList(in, xml_schema::flags::dont_validate, props)); } i used the custom build rules to define the namespace already, however i am not perfectly sure how to handle xmlns:tns. Thanks for your time,Regards,Haydar From boris at codesynthesis.com Tue Oct 28 06:51:57 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Tue Oct 28 06:59:08 2014 Subject: [xsd-users] XSD xmlns:tns issue In-Reply-To: References: Message-ID: Hi Haydar, Haydar Ozan KANTARCI writes: > props.no_namespace_schema_location("hello.xsd"); Your schema has a target namespace so you should use schema_location(). > i used the custom build rules to define the namespace already, > however i am not perfectly sure how to handle xmlns:tns. What are you trying to achieve and what exactly doesn't work? Boris From svetlana.samsonik at thomsonreuters.com Tue Oct 28 21:28:44 2014 From: svetlana.samsonik at thomsonreuters.com (svetlana.samsonik@thomsonreuters.com) Date: Tue Oct 28 21:28:55 2014 Subject: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 In-Reply-To: References: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3F49@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5D9AC8@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5EC893@C111TYXEMBX72.ERF.thomson.com> Message-ID: <1BAB72DB2329DF4CA1735854E449C0AD1D5F0126@C111TYXEMBX72.ERF.thomson.com> Hi Boris, You are absolutely right about optimized version. I did not give you correct results and yes, there is no performance difference between c++98 and c++11 optimized. I double checked non-optimized results and my numbers didn't change. --std c++98 -O0 --std c++98 -O2 --std c++11 -O0 --std c++11 -O2 Parse XML: 4715 3769 5420 3509 Serialize XML: 4998 3958 5735 3980 Serialize XRD: 325 98 329 92 Pasre XDR: 725 234 1405 230 Do you still want parse() function? My newsMessage object is deeply nested. Thank you, Svetlana -----Original Message----- From: Boris Kolpackov [mailto:boris@codesynthesis.com] Sent: Monday, October 27, 2014 9:28 AM To: Samsonik, Svetlana (TR Technology) Cc: xsd-users@codesynthesis.com Subject: Re: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 Hi Svetlana, svetlana.samsonik@thomsonreuters.com > writes: > --std c++98 -O0 --std c++98 -O2 --std c++11 -O0 --std c++11 -O2 > > Parse XML: 4715/ 3769 4667 5420 5648/ 3509 > > Serialize XML: 4998 /3958 5082 5735 6038/ 3980 > > Serialize XRD: 325 / 98 356 329 332 / 92 > > Pasre XDR: 725 / 234 765 1405 1444/ 230 Wow, these numbers are hard to believe. Especially the Parse XDR part. What they tell me are two things: 1. C++11 code is twice as slow as C++98. 2. Code built with optimization disabled (-O0) is faster than with optimization enabled (-O2). #2 is especially hard to believe. Can you maybe double check that the test setup is correct? > Both constructors are identical for 3.3.0 and 4.0.0: > > > namespace power > { > newsMessage:: > newsMessage (::xml_schema::istream< XDR >& s, > ::xml_schema::flags f, > ::xml_schema::container* c) > : ::xml_schema::type (s, f, c), > header_ (this), > itemSet_ (this) > { > this->parse (s, f); > } Can you also send the parse() function bodies as well (the ones that take the XDR stream)? Boris From boris at codesynthesis.com Wed Oct 29 06:50:00 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Oct 29 06:57:11 2014 Subject: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 In-Reply-To: <1BAB72DB2329DF4CA1735854E449C0AD1D5F0126@C111TYXEMBX72.ERF.thomson.com> References: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3F49@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5D9AC8@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5EC893@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5F0126@C111TYXEMBX72.ERF.thomson.com> Message-ID: Hi Svetlana, svetlana.samsonik@thomsonreuters.com writes: > I did not give you correct results and yes, there is no performance > difference between c++98 and c++11 optimized. > > --std c++98 -O0 --std c++98 -O2 --std c++11 -O0 --std c++11 -O2 > > Pasre XDR: 725 234 1405 230 Ok, this makes much more sense now. While it is curious that the non- optimized C++11 build is twice as slow as non-optimized C++98, it is not entirely unexpected. Talking about performance of non-optimized code doesn't really make much sense; all kind of things can be going on there (e.g., assert's, debug checks, etc). > Do you still want parse() function? My newsMessage object is deeply nested. In my view this is resolved unless you for some reason expect better performance of the debug builds. If you are also satisfied with the result, then, no, I don't need the parse() functions. Boris From svetlana.samsonik at thomsonreuters.com Wed Oct 29 07:47:25 2014 From: svetlana.samsonik at thomsonreuters.com (svetlana.samsonik@thomsonreuters.com) Date: Wed Oct 29 07:47:51 2014 Subject: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 In-Reply-To: References: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3F49@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5D9AC8@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5EC893@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5F0126@C111TYXEMBX72.ERF.thomson.com> Message-ID: <1BAB72DB2329DF4CA1735854E449C0AD1D5F0C05@C111TYXEMBX72.ERF.thomson.com> Hi Boris, Optimized results are completely satisfactory. Thanks a lot for your help! Svetlana -----Original Message----- From: Boris Kolpackov [mailto:boris@codesynthesis.com] Sent: Wednesday, October 29, 2014 6:50 AM To: Samsonik, Svetlana (TR Technology) Cc: xsd-users@codesynthesis.com Subject: Re: [xsd-users] Binary serialization performance in version 4.0.0 vs 3.3.0 Hi Svetlana, svetlana.samsonik@thomsonreuters.com writes: > I did not give you correct results and yes, there is no performance > difference between c++98 and c++11 optimized. > > --std c++98 -O0 --std c++98 -O2 --std c++11 -O0 --std c++11 -O2 > > Pasre XDR: 725 234 1405 230 Ok, this makes much more sense now. While it is curious that the non- optimized C++11 build is twice as slow as non-optimized C++98, it is not entirely unexpected. Talking about performance of non-optimized code doesn't really make much sense; all kind of things can be going on there (e.g., assert's, debug checks, etc). > Do you still want parse() function? My newsMessage object is deeply nested. In my view this is resolved unless you for some reason expect better performance of the debug builds. If you are also satisfied with the result, then, no, I don't need the parse() functions. Boris From svetlana.samsonik at thomsonreuters.com Wed Oct 29 18:06:49 2014 From: svetlana.samsonik at thomsonreuters.com (svetlana.samsonik@thomsonreuters.com) Date: Wed Oct 29 18:07:09 2014 Subject: [xsd-users] Missing include in version 4.0.0 In-Reply-To: <1BAB72DB2329DF4CA1735854E449C0AD1D5D3AFB@C111TYXEMBX72.ERF.thomson.com> References: <1BAB72DB2329DF4CA1735854E449C0AD1D5C3F3D@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5C52C2@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5D3AFB@C111TYXEMBX72.ERF.thomson.com> Message-ID: <1BAB72DB2329DF4CA1735854E449C0AD1D5F1A93@C111TYXEMBX72.ERF.thomson.com> Hi Boris, Is version 4.1.0 going to be released in the near future? Thank you, Svetlana -----Original Message----- From: xsd-users-bounces@codesynthesis.com [mailto:xsd-users-bounces@codesynthesis.com] On Behalf Of Samsonik, Svetlana (TR Technology) Sent: Wednesday, October 15, 2014 11:36 AM To: xsd-users@codesynthesis.com Subject: RE: [xsd-users] Missing include in version 4.0.0 Hi Boris, No issues found with the pre-release library. Thank you, Svetlana -----Original Message----- From: Boris Kolpackov [mailto:boris@codesynthesis.com] Sent: Tuesday, October 07, 2014 2:59 AM To: Samsonik, Svetlana (TR Technology) Cc: xsd-users@codesynthesis.com Subject: Re: [xsd-users] Missing include in version 4.0.0 Hi Svetlana, svetlana.samsonik@thomsonreuters.com writes: > Yes, I would like a pre-release binary. Here you go: http://codesynthesis.com/~boris/tmp/xsd/xsd-4.1.0.a4-i686-linux-gnu.tar.bz2 Let me know if there are any issues. Boris From boris at codesynthesis.com Thu Oct 30 02:03:01 2014 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Oct 30 02:10:12 2014 Subject: [xsd-users] Missing include in version 4.0.0 In-Reply-To: <1BAB72DB2329DF4CA1735854E449C0AD1D5F1A93@C111TYXEMBX72.ERF.thomson.com> References: <1BAB72DB2329DF4CA1735854E449C0AD1D5C3F3D@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5C52C2@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5D3AFB@C111TYXEMBX72.ERF.thomson.com> <1BAB72DB2329DF4CA1735854E449C0AD1D5F1A93@C111TYXEMBX72.ERF.thomson.com> Message-ID: Hi Svetlana, svetlana.samsonik@thomsonreuters.com writes: > Is version 4.1.0 going to be released in the near future? There is a plan to release 4.0.1 bugfix-only release before year end. Boris From haydar.kantarci at hotmail.com Fri Oct 31 08:46:00 2014 From: haydar.kantarci at hotmail.com (Haydar Ozan KANTARCI) Date: Fri Oct 31 08:46:07 2014 Subject: [xsd-users] RE: xsd-users Digest, Vol 111, Issue 15 In-Reply-To: <201410281600.s9SG04uD027172@codesynthesis.com> References: <201410281600.s9SG04uD027172@codesynthesis.com> Message-ID: Hello Boris, I am currently trying to convert my XSD project from VS2008 to QT Creator. I had difficulties with namespace configurations, but was able to overcome with your help. Now, there is a problem with the parsing of Arabic characters from my XML file.