Tada! A new serializer is coming!
-
Super Lloyd wrote:
I might make a CodeProject article
no not might - you must write an article! :-D
Dave Find Me On:Web|Facebook|Twitter|LinkedIn Folding Stats: Team CodeProject
Alright then! :)
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Congratulations on your breakthrough ! I do wonder if you have really explored systematically what the WCF Serialization/Deserialization tools (DataContract, DataMember, etc.) can do. For example, I can serialize/deserialize a WinForm TreeView TreeNodeCollection object by providing the necessary Type declarations and 'KnownType Attributes. I intend to publish this technique soon, since I have not seen it before (which doesn't mean it isn't "out there"). There is also a remarkable serializer presented in an article here on CodeProject, by Christophe Bertrand (2015), that is the only serializer I've seen that can serialize WinForm Controls as is [1] : [^]. [1] "as is:" i.e., not by creating a kind of "shadow class" that serializes only certain Properties, Fields, etc.
«There is a spectrum, from "clearly desirable behaviour," to "possibly dodgy behavior that still makes some sense," to "clearly undesirable behavior." We try to make the latter into warnings or, better, errors. But stuff that is in the middle category you don’t want to restrict unless there is a clear way to work around it.» Eric Lippert, May 14, 2008
Yes, yes I did! One the problem with
KnownTypeAttributeis that it disallow plugin, code that are not know when the base class is written. Alright then, what aboutDataContractResolverthen? Well it helps... but one still need to putDataContractandDataMemberattribute on everything that need be serialized otherwise it will be (sometimes) omitted. Further as we used it at work, we kept stumbling on bug where some (new) subclass where not serialized properly for lack of proper DataContract and DataMember annotation. To summarise the second problem withDataContractSerializer, it is far from idiot proof! On thatNewtonsoft.Jsonseems like a better deal. The development constraints it imposes are minimal and, with some settings, it supports circular reference and strongly typed serialization. However it is way too chatty! Finally, using easy to define surrogate classes I can even serialize such things as Stream, Bitmap, etc... (I did define DateTime, TimeSpan, Tuple<1,2...> surrogates) And it also supports IList, IList<T>. IDictionary, IDictionary<K, V> Contrast that with one of my test method:public class AAA { public int ID { get; set; } public string Name { get; set; } public List Heights { get; set; } public List PreviousHeights { get; set; } public DayOfWeek Day { get; set; } public AAA Child { get; set; } public Tuple Tuple { get; set; } public List<object> Objects { get; set; } = new List<object>(); } \[Fact\] public void SimpleSerialTest() { // create object to be serialized var aaa = new AAA() { ID = 73, Name = "hello", Heights = new List { 1, 2, 4, 8, 16 }, Day = DayOfWeek.Sunday, Tuple = Tuple.Create(42, "42"), }; aaa.Child = aaa; aaa.Objects.Add(aaa); aaa.Objects.Add("hello"); // write it to a string build (MemoryStream alternative exists) var sb = new StringBuilder(); var pw = new PrimitiveTextWriter(new StringWriter(sb)); var ow = new ObjectWriter(pw); ow.Write(aaa); // read it var s = sb.ToString(); var pr = n -
Congratulations on your breakthrough ! I do wonder if you have really explored systematically what the WCF Serialization/Deserialization tools (DataContract, DataMember, etc.) can do. For example, I can serialize/deserialize a WinForm TreeView TreeNodeCollection object by providing the necessary Type declarations and 'KnownType Attributes. I intend to publish this technique soon, since I have not seen it before (which doesn't mean it isn't "out there"). There is also a remarkable serializer presented in an article here on CodeProject, by Christophe Bertrand (2015), that is the only serializer I've seen that can serialize WinForm Controls as is [1] : [^]. [1] "as is:" i.e., not by creating a kind of "shadow class" that serializes only certain Properties, Fields, etc.
«There is a spectrum, from "clearly desirable behaviour," to "possibly dodgy behavior that still makes some sense," to "clearly undesirable behavior." We try to make the latter into warnings or, better, errors. But stuff that is in the middle category you don’t want to restrict unless there is a clear way to work around it.» Eric Lippert, May 14, 2008
Thanks for the link, btw! Will see if I can get some idea from it! :D Must try to serialize a WPF app hey! good idea! Also.. My default setting for now is to save public property and fields. I wonder if it's the right default settings.. might find some insight in that other serializer... This serializer seems damn good too! Will have to check mine compare to his... But.. I can aleady see I have more work to do to deserialize a windows form app! :~ :laugh:
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Yes, yes I did! One the problem with
KnownTypeAttributeis that it disallow plugin, code that are not know when the base class is written. Alright then, what aboutDataContractResolverthen? Well it helps... but one still need to putDataContractandDataMemberattribute on everything that need be serialized otherwise it will be (sometimes) omitted. Further as we used it at work, we kept stumbling on bug where some (new) subclass where not serialized properly for lack of proper DataContract and DataMember annotation. To summarise the second problem withDataContractSerializer, it is far from idiot proof! On thatNewtonsoft.Jsonseems like a better deal. The development constraints it imposes are minimal and, with some settings, it supports circular reference and strongly typed serialization. However it is way too chatty! Finally, using easy to define surrogate classes I can even serialize such things as Stream, Bitmap, etc... (I did define DateTime, TimeSpan, Tuple<1,2...> surrogates) And it also supports IList, IList<T>. IDictionary, IDictionary<K, V> Contrast that with one of my test method:public class AAA { public int ID { get; set; } public string Name { get; set; } public List Heights { get; set; } public List PreviousHeights { get; set; } public DayOfWeek Day { get; set; } public AAA Child { get; set; } public Tuple Tuple { get; set; } public List<object> Objects { get; set; } = new List<object>(); } \[Fact\] public void SimpleSerialTest() { // create object to be serialized var aaa = new AAA() { ID = 73, Name = "hello", Heights = new List { 1, 2, 4, 8, 16 }, Day = DayOfWeek.Sunday, Tuple = Tuple.Create(42, "42"), }; aaa.Child = aaa; aaa.Objects.Add(aaa); aaa.Objects.Add("hello"); // write it to a string build (MemoryStream alternative exists) var sb = new StringBuilder(); var pw = new PrimitiveTextWriter(new StringWriter(sb)); var ow = new ObjectWriter(pw); ow.Write(aaa); // read it var s = sb.ToString(); var pr = nI really appreciate your lengthy reply, and your comments on your experience with WCF DataContract. I'd like to join in with the other comments here, and express my hope you will write a CP article on your work ! fyi: the XML output of the WCF DataContract serializer is "bloated" with, imho, bizarrely long internal field/tag-names: that's why I have gotten in the habit of GZiping it going and coming ... typical reduction in file size greater than 70% in my experience. imho, one of the best "acid tests" for a serializer is self-referential types like a TreeNode that contains a pointer to its Parent Node, a list of pointers to its child nodes ... and, in the case of WinForms TreeView/TreeNodes, a pointer to the TreeView Control itself. cheers, Bill
«There is a spectrum, from "clearly desirable behaviour," to "possibly dodgy behavior that still makes some sense," to "clearly undesirable behavior." We try to make the latter into warnings or, better, errors. But stuff that is in the middle category you don’t want to restrict unless there is a clear way to work around it.» Eric Lippert, May 14, 2008
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
Super Lloyd wrote:
since I shared plenty of reflection extension method....
An article in itself. I've been having fun with reflection and LINQ, but I'd really like to see some cool reflection extension methods, as well as that serializer that you describe. Marc
Imperative to Functional Programming Succinctly Contributors Wanted for Higher Order Programming Project! Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny
-
Super Lloyd wrote:
since I shared plenty of reflection extension method....
An article in itself. I've been having fun with reflection and LINQ, but I'd really like to see some cool reflection extension methods, as well as that serializer that you describe. Marc
Imperative to Functional Programming Succinctly Contributors Wanted for Higher Order Programming Project! Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny
Not something which struck my mind initially... But now that you mention I have handy methods to try to call a constructor or a method... using a list of parameters... and they match using default value as well! Which I haven't seen very often! (using LINQ magic it's just one BIG statement! haha!) Might do a tips / trick for those instead.. less work! :laugh: Other than those astute methods (TryConstruct() TryInvoke() TryConvert()) the rest is various very simple method to plug the hole in the PCL version of Reflection.
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
Versioning support: Can it read a stream written by an older version, with well-defined and useful behavior for the properties not present in the stream? Can it read a stream written by a newer version that contains properties it does not understand? Another problem I've commonly ran into is class invariants: they must be guaranteed when the class is properly constructed, but during serialization, it is not. Typically, they are enforced through getters/setters - but if a serializer initializes an object through those, it can encounter incorrect intermediate states. If constraints are not checked during read, then you need a separate function validating the object, duplicating the code in the getters / setters. I haven't encountered a enjoyable solution to that problem that did not end with "well, then you have to change how your class is implemented". Or, to snowclone a wonderful quote: The amount of serializers available is a sure sign that none of them really works.
-
Super Lloyd wrote:
since I shared plenty of reflection extension method....
An article in itself. I've been having fun with reflection and LINQ, but I'd really like to see some cool reflection extension methods, as well as that serializer that you describe. Marc
Imperative to Functional Programming Succinctly Contributors Wanted for Higher Order Programming Project! Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny
In fact I also have another serialization utility. I didn't think of it since it has nothing to do with the serializer. But it let you write something this handy!
doNotCollect = PropertyPath.Watch(aValue, x => x.A.B.C.D, newD => { /* do something */ });
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Versioning support: Can it read a stream written by an older version, with well-defined and useful behavior for the properties not present in the stream? Can it read a stream written by a newer version that contains properties it does not understand? Another problem I've commonly ran into is class invariants: they must be guaranteed when the class is properly constructed, but during serialization, it is not. Typically, they are enforced through getters/setters - but if a serializer initializes an object through those, it can encounter incorrect intermediate states. If constraints are not checked during read, then you need a separate function validating the object, duplicating the code in the getters / setters. I haven't encountered a enjoyable solution to that problem that did not end with "well, then you have to change how your class is implemented". Or, to snowclone a wonderful quote: The amount of serializers available is a sure sign that none of them really works.
Hey, The intent of this serializer is to replace
JsonSerializer(NOT the .NET Serializer) (i.e it's a document serialize, won't work on WPF/WinForm class, unfortunately...) while being way more compact, supportIList, IDictionary(normal and generic version), be strongly typed yet type error tolerant. I also dropped human readability format. Text output is just for (painful) debugging purpose. It also DOES NOT support delegates. *It is, indeed, not perfect....* Also it doesn't supportTypeConverter / ValueConverter / ISerializable, but I am thinking to add support for them... So to address each point in detail: - by default it only serialize public fields and property. This can be modified with attribute on the class and/or field/property. - what about versioning? It doesn't care if the property is missing while deserializing it is ignored - class invariant... Right now one can enumerate all **reference** object deserialized with the ObjectContext of the deserializer. **I was thinking / maybe** to go one step further and create anIDeserializedinterface to automatically "awake / complete" objects after deserialization - also difficult type to serialize can be helped with an "easy" to implementISurrogate<>class, define this class to replace problematic type when serializing / deserializingpublic interface ISurrogate {
void Initialize(T value); // called when serializing
T Instance(); // called when deserializing
}- thinking of WPF.. I just realized that some readonly property (such as
ChildrenUI component) while being readonly, can still be serialized as anIList, should be fixed by publication time! ;) Forget that, it cause too much ambiguity about the serizalizer, since it can also (optionally) serialize private field.All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
Really nice SL !
-
Really nice SL !
Haha, thanks! ;P
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
One question pops out though: why not protobuf.net? Protocol buffers provide extremely space efficient binary files, with versioning support, and .NET port can use reflection to configure the serializer without attributes. It's also among fastest serializers out there.
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Yes, yes I did! One the problem with
KnownTypeAttributeis that it disallow plugin, code that are not know when the base class is written. Alright then, what aboutDataContractResolverthen? Well it helps... but one still need to putDataContractandDataMemberattribute on everything that need be serialized otherwise it will be (sometimes) omitted. Further as we used it at work, we kept stumbling on bug where some (new) subclass where not serialized properly for lack of proper DataContract and DataMember annotation. To summarise the second problem withDataContractSerializer, it is far from idiot proof! On thatNewtonsoft.Jsonseems like a better deal. The development constraints it imposes are minimal and, with some settings, it supports circular reference and strongly typed serialization. However it is way too chatty! Finally, using easy to define surrogate classes I can even serialize such things as Stream, Bitmap, etc... (I did define DateTime, TimeSpan, Tuple<1,2...> surrogates) And it also supports IList, IList<T>. IDictionary, IDictionary<K, V> Contrast that with one of my test method:public class AAA { public int ID { get; set; } public string Name { get; set; } public List Heights { get; set; } public List PreviousHeights { get; set; } public DayOfWeek Day { get; set; } public AAA Child { get; set; } public Tuple Tuple { get; set; } public List<object> Objects { get; set; } = new List<object>(); } \[Fact\] public void SimpleSerialTest() { // create object to be serialized var aaa = new AAA() { ID = 73, Name = "hello", Heights = new List { 1, 2, 4, 8, 16 }, Day = DayOfWeek.Sunday, Tuple = Tuple.Create(42, "42"), }; aaa.Child = aaa; aaa.Objects.Add(aaa); aaa.Objects.Add("hello"); // write it to a string build (MemoryStream alternative exists) var sb = new StringBuilder(); var pw = new PrimitiveTextWriter(new StringWriter(sb)); var ow = new ObjectWriter(pw); ow.Write(aaa); // read it var s = sb.ToString(); var pr = nYour serializer looks quite easy to use. The output is human readable, but does not look to be human 'intuitive'. To get the data even more compact, maybe nice to have an option to compress (GZipStream?) the output since it does not look like anyone would ever 'hand edit' the serialized output, correct?? What am I missing? Look forward to your CP article! Curtis
-
Nah... While I be happy to be popular with it, I won't deny it! :cool: Its primary aim is to provide a much needed help with super secret project (where JSON just won't do)...
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
One question pops out though: why not protobuf.net? Protocol buffers provide extremely space efficient binary files, with versioning support, and .NET port can use reflection to configure the serializer without attributes. It's also among fastest serializers out there.
Well... ignorance perhaps? I am only proficient with the .NET Serializer, DataContractSerializer, XmlSerializer and JsonSerializer. None of them satisfied me... So a few question about protobuf: 1. can it save IList, IList<T>, IDIctionary, IDictionary<K, V> ? 2. can it provide a simple mechanism (like surrogate) to serialize opaque type like Bitmap? 3. can it cope with a class which have different property altogether after serialization? 3.a. how does it do (3)? My serializer do 3 by saving class meta data once so that it knows what to read... does it do the same or does it save like JSON: ('property name' 'property value'), one my main contention with JSON... my serializer save type metada once, then only ('property value' 'property value' 'property value'......)
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Haha thanks... I had a quick look at Transit.. but it's not C#! All my super secret take over the world project are in C#...
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Your serializer looks quite easy to use. The output is human readable, but does not look to be human 'intuitive'. To get the data even more compact, maybe nice to have an option to compress (GZipStream?) the output since it does not look like anyone would ever 'hand edit' the serialized output, correct?? What am I missing? Look forward to your CP article! Curtis
The Serializer write to an interface I created call
IPrimitiveWriterI provide 2 implementation of IPrimitiveWriterpublic class PrimitiveBinaryWriter : IPrimitiveWriter
{
public PrimitiveBinaryWriter(Stream stream)
}
public class PrimitiveTextWriter : IPrimitiveWriter
{
public PrimitiveTextWriter(TextWriter writer)
}So... binary output is covered. And it's also forward only (unlike first version, haha), so the stream can be a GZipStream. The text output is there mostly for fun. It also provide a tiny help for debugging and help me check there is no repetition (all reference, including string should be there only once)
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
The Serializer write to an interface I created call
IPrimitiveWriterI provide 2 implementation of IPrimitiveWriterpublic class PrimitiveBinaryWriter : IPrimitiveWriter
{
public PrimitiveBinaryWriter(Stream stream)
}
public class PrimitiveTextWriter : IPrimitiveWriter
{
public PrimitiveTextWriter(TextWriter writer)
}So... binary output is covered. And it's also forward only (unlike first version, haha), so the stream can be a GZipStream. The text output is there mostly for fun. It also provide a tiny help for debugging and help me check there is no repetition (all reference, including string should be there only once)
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
Great! Looking forward to trying it after your CP Article.....