From ivan.lelann at free.fr Tue Sep 7 11:53:55 2010 From: ivan.lelann at free.fr (Ivan Le Lann) Date: Tue Sep 7 12:00:13 2010 Subject: [xsde-users] Re: [xsd-users] Using a SAX parser in cxx-tree In-Reply-To: <190153702.1476961283874656639.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: <804809057.1477221283874835705.JavaMail.root@zimbra36-e6.priv.proxad.net> Hi Boris, ----- "Boris Kolpackov" a ?crit : > If your schema doesn't have a target namespace, then you should use > this > version of the document_simpl constructor: > > xml_schema::document_simpl doc_s ( > catalog_s.root_serializer (), > catalog_s.root_name ()); > I'm sorry that I did not try this myself. Thanks again. Nevertheless, it seems to me that the patch I suggested still makes sense. Regards, Ivan. From ivan.lelann at free.fr Tue Sep 7 13:24:44 2010 From: ivan.lelann at free.fr (Ivan Le Lann) Date: Tue Sep 7 16:32:50 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <614640443.1482391283879986542.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: <833352866.1482611283880284116.JavaMail.root@zimbra36-e6.priv.proxad.net> Hi all, I played with XSDe/Hybrid 3.1 and I'm unsurprisingly very happy with it. I do have one comment about "variable-length" (VL) elements. The combination of these two facts : 1) VL elements have no copy constructors. 2) an element takes single ownership of its VL child elements. does not help when you want to assemble an object from cached subparts. It seems you're stuck to rebuild/reparse your subpart on each member setter call. While I do understand the move against deep copy, I see no "non destructive" shallow copy alternative. Does anyone know an existing way to handle such "object assembling" with xsde ? So far, the solutions I see would be to either : 1) use copy-on-write or shared pointer in containers of some user-listed VL types. 2) allow deep copy of some user-listed VL types. At first sight, both ways imply a change in xsde compiler code, and in non-generated libxsde/hybrid sequences templates. While I would be happy to try one of these, I'm not sure I can call it a solution. Regards, Ivan. From boris at codesynthesis.com Wed Sep 8 10:27:06 2010 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Sep 8 10:15:11 2010 Subject: [xsde-users] Re: [xsd-users] Using a SAX parser in cxx-tree In-Reply-To: <804809057.1477221283874835705.JavaMail.root@zimbra36-e6.priv.proxad.net> References: <190153702.1476961283874656639.JavaMail.root@zimbra36-e6.priv.proxad.net> <804809057.1477221283874835705.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: Hi Ivan, Ivan Le Lann writes: > Nevertheless, it seems to me that the patch I suggested still makes sense. Yes, probably. I, however, fixed this in a different way: http://scm.codesynthesis.com/?p=xsde/xsde.git;a=commit;h=d6313f9155195b856ee8170efb6d67e4072e64bc Boris From boris at codesynthesis.com Wed Sep 8 10:40:21 2010 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Sep 8 10:28:20 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <833352866.1482611283880284116.JavaMail.root@zimbra36-e6.priv.proxad.net> References: <614640443.1482391283879986542.JavaMail.root@zimbra36-e6.priv.proxad.net> <833352866.1482611283880284116.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: Hi Ivan, Ivan Le Lann writes: > I played with XSDe/Hybrid 3.1 and I'm unsurprisingly very happy with it. I am glad you are enjoying it. > While I do understand the move against deep copy, > I see no "non destructive" shallow copy alternative. > Does anyone know an existing way to handle such "object assembling" > with xsde ? There is currently no support for this. If you need to copy the var-length object, you need to manually copy its members as well. The alternative could be to serialize to XML using several parts instead of one object model thus avoiding copying altogether (here I assume that the goal of constructing an object from multiple parts it to eventually write it out to XML). > So far, the solutions I see would be to either : > > 1) use copy-on-write or shared pointer in containers of some user-listed > VL types. I think COW or reference counting would be too heavy-weight for XSD/e. Remember that its primary targets are mobile and embedded systems which can have only very basic C++ support, without exceptions, etc. Implementing error handling in COW without exceptions could be challenging. Also the overhead of storing a counter in each var-length object could be too high for some memory-constrained systems. > 2) allow deep copy of some user-listed VL types. That's on the TODO for the next release. There will be an option to request generating of copy constructors/clone functions for var-length types. Boris From ivan.lelann at free.fr Wed Sep 8 11:20:10 2010 From: ivan.lelann at free.fr (Ivan Le Lann) Date: Wed Sep 8 11:20:21 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <634576022.1540101283959159393.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: <608260859.1540151283959210124.JavaMail.root@zimbra36-e6.priv.proxad.net> Hi Boris, ----- "Boris Kolpackov" a ?crit : > The alternative could be to serialize to XML using several parts > instead of one object model thus avoiding copying altogether (here > I assume that the goal of constructing an object from multiple > parts it to eventually write it out to XML). I might end with something like this, but I wanted to avoid it. > > So far, the solutions I see would be to either : > > > > 1) use copy-on-write or shared pointer in containers of some > > user-listed VL types. > > I think COW or reference counting would be too heavy-weight for > XSD/e. Remember that its primary targets are mobile and embedded > systems which can have only very basic C++ support, without > exceptions, etc. > Implementing error handling in COW without exceptions could be > challenging. Also the overhead of storing a counter in each > var-length object could be too high for some memory-constrained > systems. What I had in mind was to "type trait" the use of pointer in XSDe/Hybrid generated code. It would then be user's responsability to specialize the trait with "--hxx-prologue" file for the types of its choice. He could also hack a libxsde header if he wants to change the default. With a default targeting T*, I thought everybody would be happy. The "You pay for what you use" axiom is respected. I actually hacked one hour in that direction and it seems easier than I first thought, at least in xsde/hybrid compiler code. It might be more tricky to end with simple C++ in libxsde/hybrid sequences. What do you think about this ? > > 2) allow deep copy of some user-listed VL types. > > That's on the TODO for the next release. There will be an option to > request generating of copy constructors/clone functions for > var-length types. > Great. :-) Regards, Ivan. > Boris From ivan.lelann at free.fr Fri Sep 10 04:31:49 2010 From: ivan.lelann at free.fr (Ivan Le Lann) Date: Fri Sep 10 04:31:58 2010 Subject: [xsde-users] variable length member : suggestion for "_present (bool x)" methods Message-ID: <589484267.1658471284107509728.JavaMail.root@zimbra36-e6.priv.proxad.net> Hi all, It seems to me that working with variable-length elements is not as transparent as it could be. Editing an element schema, with C++ code going from fix to variable length, you might silently break some code. Typically. // foo f; f.bar_present(true); f.bar().baz(); // If "bar" becomes variable length some day, you're dead. I propose to add an option to change generated code in "foo": inline void foo:: bar_present (bool x) { if (!x) { delete this->bar_; this->bar_ = 0; } // patch suggestion : else if (!bar_present()) { this->bar_ = new bar; } } If by any chance this suggestion has some success, I would be glad to provide a patch. Regards, Ivan. From boris at codesynthesis.com Fri Sep 10 08:49:48 2010 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Sep 10 08:37:10 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <608260859.1540151283959210124.JavaMail.root@zimbra36-e6.priv.proxad.net> References: <634576022.1540101283959159393.JavaMail.root@zimbra36-e6.priv.proxad.net> <608260859.1540151283959210124.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: Hi Ivan, Ivan Le Lann writes: > What I had in mind was to "type trait" the use of pointer in XSDe/Hybrid > generated code. It would then be user's responsability to specialize > the trait with "--hxx-prologue" file for the types of its choice. > He could also hack a libxsde header if he wants to change the default. > With a default targeting T*, I thought everybody would be happy. > The "You pay for what you use" axiom is respected. > > I actually hacked one hour in that direction and it seems easier than > I first thought, at least in xsde/hybrid compiler code. > It might be more tricky to end with simple C++ in libxsde/hybrid sequences. > > What do you think about this ? Template specialization, and especially partial template specialization, is unfortunately off limits in XSD/e. We still need to support g++ 2.95 (VxWorks 5.x) and eVC++ 4.0 (WinCE). eVC++ 4.0 is pretty much VC++ 6 when it comes to C++ language conformance. But the idea is not bad. In fact, I used this approach in another project I am working on (not yet released). One drawback is the more obscure interface. Compare: string* clone (); to: pointer_traits::type clone (); Boris From boris at codesynthesis.com Fri Sep 10 09:10:54 2010 From: boris at codesynthesis.com (Boris Kolpackov) Date: Fri Sep 10 08:58:16 2010 Subject: [xsde-users] variable length member : suggestion for "_present (bool x)" methods In-Reply-To: <589484267.1658471284107509728.JavaMail.root@zimbra36-e6.priv.proxad.net> References: <589484267.1658471284107509728.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: Hi Ivan, Ivan Le Lann writes: > Editing an element schema, with C++ code going from fix to variable > length, you might silently break some code. Typically. > > // > foo f; > f.bar_present(true); > f.bar().baz(); > // > > If "bar" becomes variable length some day, you're dead. > I propose to add an option to change generated code in "foo": > > inline > void foo:: > bar_present (bool x) > { > if (!x) > { > delete this->bar_; > this->bar_ = 0; > } > > // patch suggestion : > else if (!bar_present()) > { > this->bar_ = new bar; > } > } Two problems with this suggestion: 1. When exceptions are disabled the memory allocation failure should be propagated to the caller somehow. Right now the generated classes carefully avoid making any (implicit) memory allocations. All var- length objects are allocated explicitly by the user. 2. The more serious problem is the fact that the newly created object (bar in the code above) is itself uninitialized. Since it is var- length, it will most likely contain members that need initialization (e.g., a string member when STL is disabled). So the patch only postpones the problem for a little while: f.bar ().baz (); // ok f.bar ().baz ().str () // segfault I agree it would be nice to help the user here somehow. I just don't see any good way to do it. Maybe not generate _present(bool) for var- length members at all? The reset behavior can be achieved by simply passing the NULL pointer to the modifier function. What do you think? Boris From ivan.lelann at free.fr Fri Sep 10 09:28:50 2010 From: ivan.lelann at free.fr (Ivan Le Lann) Date: Fri Sep 10 09:29:00 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <1489198963.1679721284124979358.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: <206932947.1680261284125330680.JavaMail.root@zimbra36-e6.priv.proxad.net> Hi Boris, > > Template specialization, and especially partial template > specialization, > is unfortunately off limits in XSD/e. We still need to support g++ > 2.95 > (VxWorks 5.x) and eVC++ 4.0 (WinCE). eVC++ 4.0 is pretty much VC++ 6 > when it comes to C++ language conformance. > XSD/e does not need template specialization to wrap its pointer code. My code is (alpha-)finished, and I think there is no such thing. One *user* needs template specialization to *use* the power provided by this wrapping. That's what I meant by "You pay for what you use". eVC++ 4.0 users would not complain just because they cannot tweak as far as they could with a decent compiler, would they ? There are a lot of other things they cannot do with generated code, and they don't blame XSD/e for that. > One drawback is the more obscure interface. I have to agree on that. Pointer encapsulation could easily be an option though. Anyway, if you allow me to, I will send you a patch against XSD/e 3.1. It is up to you to choose what you want to do with it. :-) Regards, Ivan. From ivan.lelann at free.fr Fri Sep 10 10:22:45 2010 From: ivan.lelann at free.fr (Ivan Le Lann) Date: Fri Sep 10 10:22:54 2010 Subject: [xsde-users] variable length member : suggestion for "_present (bool x)" methods In-Reply-To: <1583496137.1683351284127480234.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: <1947904879.1684731284128565223.JavaMail.root@zimbra36-e6.priv.proxad.net> Hi Boris, I hope I do not bother you with all my comments! ----- "Boris Kolpackov" a ?crit : > > Two problems with this suggestion: > > 1. When exceptions are disabled I definitely overlooked the exception point. "_present (bool)" could be switch to void/error, but that's quite ugly. > 2. The more serious problem is the fact that the newly created object > (bar in the code above) is itself uninitialized. Since it is var- > length, it will most likely contain members that need > initialization > (e.g., a string member when STL is disabled). So the patch only > postpones the problem for a little while: > > f.bar ().baz (); // ok > f.bar ().baz ().str () // segfault > Please let me stress that my point is not to avoid all segfaults. It is to be able to write the same code for fix/var elements, as much as possible. I am focusing on "_present" but as you suggest, there might be other harder cases to harmonize, like simple accessors. While "_prevent(false)" is ok, since it deallocates on non-null pointer, "_present(true)" is not since it does not allocate on null pointer . Two solutions to adapt my suggestion : 1) change the return type of _present(bool) to void/error 2) isolate new behavior in a "void/error _check_present()" >Maybe not generate _present(bool) for var-length members at all? You don't end with var/fix independant code, but at least, this is compile time error. I consider it a better way. This is the exact opposite of what I suggested, though! :-D As for elements accessors without "_present" flag, I indeed see no obvious solution that would fit with "no exception" support. Ivan From ivan.lelann at free.fr Mon Sep 13 04:49:36 2010 From: ivan.lelann at free.fr (Ivan Le Lann) Date: Mon Sep 13 04:49:47 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <206932947.1680261284125330680.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: <1889732707.1803691284367776249.JavaMail.root@zimbra36-e6.priv.proxad.net> Hi Boris, Quoting myself : > eVC++ 4.0 users would not complain just because they cannot tweak > as far as they could with a decent compiler, would they ? I forgot to mention that even users of poor C++ compilers could still change the default pointer behavior by editing libxsde. What they would not be able to do is to change pointer behavior only for specific types. The only way to allow them to do so would be to forget type traits and rather extend the command line. Something like extending "--custom-type", to directly generate the good pointer name instead of type traits, and handle headers inclusion somehow. Libxsde template sequences would then need a typedef in generated types to have correct behavior. I don't know which way is the best. Maybe you dislike them equally ? :-) Ivan. From boris at codesynthesis.com Mon Sep 13 11:38:06 2010 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Sep 13 11:24:36 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <1889732707.1803691284367776249.JavaMail.root@zimbra36-e6.priv.proxad.net> References: <206932947.1680261284125330680.JavaMail.root@zimbra36-e6.priv.proxad.net> <1889732707.1803691284367776249.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: Hi Ivan, Ivan Le Lann writes: > The only way to allow them to do so would be to forget type traits and > rather extend the command line. Yes, that would probably be preferable. As general guidelines for XSD/e, we should always prefer to generate simpler, even if less elegant, code rather than rely on more advanced features such as templates. It also needs to work equally well with all possible configurations of XSD/e (STL/no-STL, exceptions/no-exceptions, etc). Also, in this particular case the ideal is no changes to the way things are currently done if the custom pointer option is not used. I think the problematic part will be var_sequence and var_iterator. Would you like to propose how we can handle them? Also, provided there is a way to clone var-length nodes, would you still need this feature? Boris From boris at codesynthesis.com Mon Sep 13 11:49:45 2010 From: boris at codesynthesis.com (Boris Kolpackov) Date: Mon Sep 13 11:36:16 2010 Subject: [xsde-users] variable length member : suggestion for "_present (bool x)" methods In-Reply-To: <1947904879.1684731284128565223.JavaMail.root@zimbra36-e6.priv.proxad.net> References: <1583496137.1683351284127480234.JavaMail.root@zimbra36-e6.priv.proxad.net> <1947904879.1684731284128565223.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: Hi Ivan, Ivan Le Lann writes: > I hope I do not bother you with all my comments! Not at all. I always appreciate feedback based on actual usage. > "_present (bool)" could be switch to void/error, but that's quite ugly. Exactly. > It is to be able to write the same code for fix/var elements, > as much as possible. I don't think this is an achievable goal in all configurations. And I don't think it makes sense to "fix" some cases and give up on the rest. It should be all or nothing. Otherwise you always have to stop and think whether this particular case is handled automatically or requires manual intervention. > >Maybe not generate _present(bool) for var-length members at all? > > You don't end with var/fix independant code, but at least, > this is compile time error. I consider it a better way. > This is the exact opposite of what I suggested, though! :-D The advantage of this approach is that it will point the location of where things need to be changed. Compare this: hoping that you didn't miss any cases where the XSD/e compiler can't handle fix/var differences vs knowing that the C++ compiler will always flag such cases. I will take the latter any day. Boris From ivan.lelann at free.fr Wed Sep 15 13:23:35 2010 From: ivan.lelann at free.fr (Ivan Le Lann) Date: Wed Sep 15 13:23:46 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <1395672243.1992031284571264723.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: <280576911.1992121284571415489.JavaMail.root@zimbra36-e6.priv.proxad.net> Hi Boris, ----- "Boris Kolpackov" writes : > Also, in this particular case the ideal is no changes to the way > things > are currently done if the custom pointer option is not used. I think > the problematic part will be var_sequence and var_iterator. Would you > like to propose how we can handle them? > In var_sequence as in generated code, we need free functions to wrap code like "p = 0" or "delete p". Without template specialization, how to implement these functions ? Overloading can help in generated code, but not in libxsde, as you spotted. This looks like a dead end to me. > Also, provided there is a way to clone var-length nodes, would you > still need this feature? > I need both to get a COW-pointer. Not that I really like COW but it seemed to fit here : The code is basically a "solid request" builder from various parts. Some parts are most of the time inserted as is, some are always, and some are rebuilt from scratch each time. My current state (with my smart ptr aware xsde) is that parts that are known never to be modified use a shared ptr. But I'm playing with fire here and especially with side-effects. Before the move to vanilla XSD/e, this code was *much* slower. This was pure DOM and XPath manipulation with TinyXML. While performance with vanilla XSD/e is a great achievement, I had the feeling I could easily be both safe and optimal with few pointer tweaks. Please note that I realize now that I'm asking much of XSD/e here. I might live with my "pointer patch" for a while, thinking of a better way to do things in the future. I might even try this idea on XSD (not /e). What do you think ? Ivan. From boris at codesynthesis.com Thu Sep 16 13:20:25 2010 From: boris at codesynthesis.com (Boris Kolpackov) Date: Thu Sep 16 13:10:23 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <280576911.1992121284571415489.JavaMail.root@zimbra36-e6.priv.proxad.net> References: <1395672243.1992031284571264723.JavaMail.root@zimbra36-e6.priv.proxad.net> <280576911.1992121284571415489.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: Hi Ivan, Ivan Le Lann writes: > In var_sequence as in generated code, we need free functions to wrap > code like "p = 0" or "delete p". Without template specialization, > how to implement these functions ? Overloading can help in generated > code, but not in libxsde, as you spotted. > > This looks like a dead end to me. Looks like it. The only alternative would be to make a copy of the var_sequence template with the second template argument which is assumed to be a smart pointer. But having essentially the same functionality in two places is quite ugly. > > Also, provided there is a way to clone var-length nodes, would you > > still need this feature? > > I need both to get a COW-pointer. I don't think you will be able to achieve COW even if there was pointer customization support. You would be able share nodes in several object models, but not COW. For COW you would need to completely encapsulate modification of a node so that you can intercept them and make a copy. For example, the generated accessor function which return a non-const reference to an object, allows me to modify the node and there is no way you will be able to intercept that. Unless you resort to something like COA (copy on access) instead. > Not that I really like COW but it seemed to fit here : > > The code is basically a "solid request" builder from various parts. > Some parts are most of the time inserted as is, some are always, > and some are rebuilt from scratch each time. > My current state (with my smart ptr aware xsde) is that parts that > are known never to be modified use a shared ptr. > But I'm playing with fire here and especially with side-effects. > > Before the move to vanilla XSD/e, this code was *much* slower. > This was pure DOM and XPath manipulation with TinyXML. > While performance with vanilla XSD/e is a great achievement, I had the > feeling I could easily be both safe and optimal with few pointer tweaks. It is not clear to me where there is COW in your logic. Sounds like just sharing of nodes among multiple object models. If that's all you need, then I might have an interesting workaround for you. For the upcoming release of XSD/e we have implemented support for custom allocators. You can basically provide your own alloc and free functions and all the dynamic memory allocated by XSD/e will go through these functions. The idea is to implement the so-called arenas where you would place your shared fragments. Your custom free() function will know about these arenas and if there is a request to free a memory block from one of them, it will simply be ignored. This way the object models can act as if they were the sole "owners" of such shared fragments when in reality your application controls when the these arenas (and therefore object model fragments placed in them) will be freed. When a fragment is no longer needed, the arena that hosts it is destroyed and all the memory allocated is freed. The only issue with this model is if you use std::string which won't be automatically allocated in the arena. But if you disable STL, then I think this approach could work quite well. > I might even try this idea on XSD (not /e). What do you think? XSD is a lot more feature-rich so there could be more places where you will have to change things. But you can definitely try. Boris From ivan.lelann at free.fr Wed Sep 29 10:21:29 2010 From: ivan.lelann at free.fr (Ivan Le Lann) Date: Wed Sep 29 10:21:40 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <529851603.2850001285767783060.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: <880333453.2853371285770089597.JavaMail.root@zimbra36-e6.priv.proxad.net> Hi Boris, Sorry for the late reply. I'm building a response for a few points you mentionned, but I do not want to rush it, so it will come later. I've just spotted a new problem : xs:choice is mapped to a union. And union members cannot have non-trivial copy constructor. That seems to prevent any smart pointer under xs:choice. I read that C++0x (at least) can handle this with some code, but that will probably not help us for XSD/e platforms. :-) Ivan. From boris at codesynthesis.com Wed Sep 29 11:57:03 2010 From: boris at codesynthesis.com (Boris Kolpackov) Date: Wed Sep 29 11:45:43 2010 Subject: [xsde-users] memory management of "variable length" elements In-Reply-To: <880333453.2853371285770089597.JavaMail.root@zimbra36-e6.priv.proxad.net> References: <529851603.2850001285767783060.JavaMail.root@zimbra36-e6.priv.proxad.net> <880333453.2853371285770089597.JavaMail.root@zimbra36-e6.priv.proxad.net> Message-ID: Hi Ivan, Ivan Le Lann writes: > I'm building a response for a few points you mentionned, but I do > not want to rush it, so it will come later. No problem. > I've just spotted a new problem : xs:choice is mapped to a union. > And union members cannot have non-trivial copy constructor. > That seems to prevent any smart pointer under xs:choice. Yes, that's a good catch. Once you start thinking about things on a deeper level, all kinds of hidden surprises pop up. > I read that C++0x (at least) can handle this with some code, > but that will probably not help us for XSD/e platforms. :-) :-) Boris