Tada! A new serializer is coming!
-
Super Lloyd wrote:
since I shared plenty of reflection extension method....
An article in itself. I've been having fun with reflection and LINQ, but I'd really like to see some cool reflection extension methods, as well as that serializer that you describe. Marc
Imperative to Functional Programming Succinctly Contributors Wanted for Higher Order Programming Project! Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny
Not something which struck my mind initially... But now that you mention I have handy methods to try to call a constructor or a method... using a list of parameters... and they match using default value as well! Which I haven't seen very often! (using LINQ magic it's just one BIG statement! haha!) Might do a tips / trick for those instead.. less work! :laugh: Other than those astute methods (TryConstruct() TryInvoke() TryConvert()) the rest is various very simple method to plug the hole in the PCL version of Reflection.
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
Versioning support: Can it read a stream written by an older version, with well-defined and useful behavior for the properties not present in the stream? Can it read a stream written by a newer version that contains properties it does not understand? Another problem I've commonly ran into is class invariants: they must be guaranteed when the class is properly constructed, but during serialization, it is not. Typically, they are enforced through getters/setters - but if a serializer initializes an object through those, it can encounter incorrect intermediate states. If constraints are not checked during read, then you need a separate function validating the object, duplicating the code in the getters / setters. I haven't encountered a enjoyable solution to that problem that did not end with "well, then you have to change how your class is implemented". Or, to snowclone a wonderful quote: The amount of serializers available is a sure sign that none of them really works.
-
Super Lloyd wrote:
since I shared plenty of reflection extension method....
An article in itself. I've been having fun with reflection and LINQ, but I'd really like to see some cool reflection extension methods, as well as that serializer that you describe. Marc
Imperative to Functional Programming Succinctly Contributors Wanted for Higher Order Programming Project! Learning to code with python is like learning to swim with those little arm floaties. It gives you undeserved confidence and will eventually drown you. - DangerBunny
In fact I also have another serialization utility. I didn't think of it since it has nothing to do with the serializer. But it let you write something this handy!
doNotCollect = PropertyPath.Watch(aValue, x => x.A.B.C.D, newD => { /* do something */ });
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Versioning support: Can it read a stream written by an older version, with well-defined and useful behavior for the properties not present in the stream? Can it read a stream written by a newer version that contains properties it does not understand? Another problem I've commonly ran into is class invariants: they must be guaranteed when the class is properly constructed, but during serialization, it is not. Typically, they are enforced through getters/setters - but if a serializer initializes an object through those, it can encounter incorrect intermediate states. If constraints are not checked during read, then you need a separate function validating the object, duplicating the code in the getters / setters. I haven't encountered a enjoyable solution to that problem that did not end with "well, then you have to change how your class is implemented". Or, to snowclone a wonderful quote: The amount of serializers available is a sure sign that none of them really works.
Hey, The intent of this serializer is to replace
JsonSerializer(NOT the .NET Serializer) (i.e it's a document serialize, won't work on WPF/WinForm class, unfortunately...) while being way more compact, supportIList, IDictionary(normal and generic version), be strongly typed yet type error tolerant. I also dropped human readability format. Text output is just for (painful) debugging purpose. It also DOES NOT support delegates. *It is, indeed, not perfect....* Also it doesn't supportTypeConverter / ValueConverter / ISerializable, but I am thinking to add support for them... So to address each point in detail: - by default it only serialize public fields and property. This can be modified with attribute on the class and/or field/property. - what about versioning? It doesn't care if the property is missing while deserializing it is ignored - class invariant... Right now one can enumerate all **reference** object deserialized with the ObjectContext of the deserializer. **I was thinking / maybe** to go one step further and create anIDeserializedinterface to automatically "awake / complete" objects after deserialization - also difficult type to serialize can be helped with an "easy" to implementISurrogate<>class, define this class to replace problematic type when serializing / deserializingpublic interface ISurrogate {
void Initialize(T value); // called when serializing
T Instance(); // called when deserializing
}- thinking of WPF.. I just realized that some readonly property (such as
ChildrenUI component) while being readonly, can still be serialized as anIList, should be fixed by publication time! ;) Forget that, it cause too much ambiguity about the serizalizer, since it can also (optionally) serialize private field.All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
Really nice SL !
-
Really nice SL !
Haha, thanks! ;P
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
One question pops out though: why not protobuf.net? Protocol buffers provide extremely space efficient binary files, with versioning support, and .NET port can use reflection to configure the serializer without attributes. It's also among fastest serializers out there.
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Yes, yes I did! One the problem with
KnownTypeAttributeis that it disallow plugin, code that are not know when the base class is written. Alright then, what aboutDataContractResolverthen? Well it helps... but one still need to putDataContractandDataMemberattribute on everything that need be serialized otherwise it will be (sometimes) omitted. Further as we used it at work, we kept stumbling on bug where some (new) subclass where not serialized properly for lack of proper DataContract and DataMember annotation. To summarise the second problem withDataContractSerializer, it is far from idiot proof! On thatNewtonsoft.Jsonseems like a better deal. The development constraints it imposes are minimal and, with some settings, it supports circular reference and strongly typed serialization. However it is way too chatty! Finally, using easy to define surrogate classes I can even serialize such things as Stream, Bitmap, etc... (I did define DateTime, TimeSpan, Tuple<1,2...> surrogates) And it also supports IList, IList<T>. IDictionary, IDictionary<K, V> Contrast that with one of my test method:public class AAA { public int ID { get; set; } public string Name { get; set; } public List Heights { get; set; } public List PreviousHeights { get; set; } public DayOfWeek Day { get; set; } public AAA Child { get; set; } public Tuple Tuple { get; set; } public List<object> Objects { get; set; } = new List<object>(); } \[Fact\] public void SimpleSerialTest() { // create object to be serialized var aaa = new AAA() { ID = 73, Name = "hello", Heights = new List { 1, 2, 4, 8, 16 }, Day = DayOfWeek.Sunday, Tuple = Tuple.Create(42, "42"), }; aaa.Child = aaa; aaa.Objects.Add(aaa); aaa.Objects.Add("hello"); // write it to a string build (MemoryStream alternative exists) var sb = new StringBuilder(); var pw = new PrimitiveTextWriter(new StringWriter(sb)); var ow = new ObjectWriter(pw); ow.Write(aaa); // read it var s = sb.ToString(); var pr = nYour serializer looks quite easy to use. The output is human readable, but does not look to be human 'intuitive'. To get the data even more compact, maybe nice to have an option to compress (GZipStream?) the output since it does not look like anyone would ever 'hand edit' the serialized output, correct?? What am I missing? Look forward to your CP article! Curtis
-
Nah... While I be happy to be popular with it, I won't deny it! :cool: Its primary aim is to provide a much needed help with super secret project (where JSON just won't do)...
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
One question pops out though: why not protobuf.net? Protocol buffers provide extremely space efficient binary files, with versioning support, and .NET port can use reflection to configure the serializer without attributes. It's also among fastest serializers out there.
Well... ignorance perhaps? I am only proficient with the .NET Serializer, DataContractSerializer, XmlSerializer and JsonSerializer. None of them satisfied me... So a few question about protobuf: 1. can it save IList, IList<T>, IDIctionary, IDictionary<K, V> ? 2. can it provide a simple mechanism (like surrogate) to serialize opaque type like Bitmap? 3. can it cope with a class which have different property altogether after serialization? 3.a. how does it do (3)? My serializer do 3 by saving class meta data once so that it knows what to read... does it do the same or does it save like JSON: ('property name' 'property value'), one my main contention with JSON... my serializer save type metada once, then only ('property value' 'property value' 'property value'......)
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Haha thanks... I had a quick look at Transit.. but it's not C#! All my super secret take over the world project are in C#...
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Your serializer looks quite easy to use. The output is human readable, but does not look to be human 'intuitive'. To get the data even more compact, maybe nice to have an option to compress (GZipStream?) the output since it does not look like anyone would ever 'hand edit' the serialized output, correct?? What am I missing? Look forward to your CP article! Curtis
The Serializer write to an interface I created call
IPrimitiveWriterI provide 2 implementation of IPrimitiveWriterpublic class PrimitiveBinaryWriter : IPrimitiveWriter
{
public PrimitiveBinaryWriter(Stream stream)
}
public class PrimitiveTextWriter : IPrimitiveWriter
{
public PrimitiveTextWriter(TextWriter writer)
}So... binary output is covered. And it's also forward only (unlike first version, haha), so the stream can be a GZipStream. The text output is there mostly for fun. It also provide a tiny help for debugging and help me check there is no repetition (all reference, including string should be there only once)
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
The Serializer write to an interface I created call
IPrimitiveWriterI provide 2 implementation of IPrimitiveWriterpublic class PrimitiveBinaryWriter : IPrimitiveWriter
{
public PrimitiveBinaryWriter(Stream stream)
}
public class PrimitiveTextWriter : IPrimitiveWriter
{
public PrimitiveTextWriter(TextWriter writer)
}So... binary output is covered. And it's also forward only (unlike first version, haha), so the stream can be a GZipStream. The text output is there mostly for fun. It also provide a tiny help for debugging and help me check there is no repetition (all reference, including string should be there only once)
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
Great! Looking forward to trying it after your CP Article.....
-
Hey, The intent of this serializer is to replace
JsonSerializer(NOT the .NET Serializer) (i.e it's a document serialize, won't work on WPF/WinForm class, unfortunately...) while being way more compact, supportIList, IDictionary(normal and generic version), be strongly typed yet type error tolerant. I also dropped human readability format. Text output is just for (painful) debugging purpose. It also DOES NOT support delegates. *It is, indeed, not perfect....* Also it doesn't supportTypeConverter / ValueConverter / ISerializable, but I am thinking to add support for them... So to address each point in detail: - by default it only serialize public fields and property. This can be modified with attribute on the class and/or field/property. - what about versioning? It doesn't care if the property is missing while deserializing it is ignored - class invariant... Right now one can enumerate all **reference** object deserialized with the ObjectContext of the deserializer. **I was thinking / maybe** to go one step further and create anIDeserializedinterface to automatically "awake / complete" objects after deserialization - also difficult type to serialize can be helped with an "easy" to implementISurrogate<>class, define this class to replace problematic type when serializing / deserializingpublic interface ISurrogate {
void Initialize(T value); // called when serializing
T Instance(); // called when deserializing
}- thinking of WPF.. I just realized that some readonly property (such as
ChildrenUI component) while being readonly, can still be serialized as anIList, should be fixed by publication time! ;) Forget that, it cause too much ambiguity about the serizalizer, since it can also (optionally) serialize private field.All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
Soudns good to me :) `IDeserialized` would allow to cope with many common problems I've encountered. `ISurrogate` could nicely isolate some complexity from the class / serializer itself. Your striked-out part reminds me of another thing: read only Collections (they are managed internally) where the elements are mutable and should be serialized. This could be handled in IDeserialized if I could construct thecollection there and then request serialization for the collection members.
-
Soudns good to me :) `IDeserialized` would allow to cope with many common problems I've encountered. `ISurrogate` could nicely isolate some complexity from the class / serializer itself. Your striked-out part reminds me of another thing: read only Collections (they are managed internally) where the elements are mutable and should be serialized. This could be handled in IDeserialized if I could construct thecollection there and then request serialization for the collection members.
Great to know IDeserialized works out for you! :) I needed such concept for my IoC+IServiceProvider to completely initialize object tree (it's called IRegistryDelegate for that). Came with that there first! :-D After all the comments I decided to add yet more functionalities and now it's all broken.... Plus, exceptionally, I work this Saturday! :(( (this my home project, not the work one) So it's gonna be delayed some more... But the current (broken) version does support ISerializable (but not [SerializableAttribute] which assume private field reading), TypeConverter (but not ValueSerializer which are not PLC / .NET Core) and it also support restoring public readonly reference property, such as:
public List Children { get; } = new List();
public Configuration Configuration { get { return existingConfig; } }I was misleaded, proved easy and helpful! :) ETA should be end of next week now...
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Great to know IDeserialized works out for you! :) I needed such concept for my IoC+IServiceProvider to completely initialize object tree (it's called IRegistryDelegate for that). Came with that there first! :-D After all the comments I decided to add yet more functionalities and now it's all broken.... Plus, exceptionally, I work this Saturday! :(( (this my home project, not the work one) So it's gonna be delayed some more... But the current (broken) version does support ISerializable (but not [SerializableAttribute] which assume private field reading), TypeConverter (but not ValueSerializer which are not PLC / .NET Core) and it also support restoring public readonly reference property, such as:
public List Children { get; } = new List();
public Configuration Configuration { get { return existingConfig; } }I was misleaded, proved easy and helpful! :) ETA should be end of next week now...
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
Super Lloyd wrote:
So it's gonna be delayed some more...
No worries, I'll be on vacation for a week. :rolleyes: (I expect it on my desk when I return, 9am point ;P )
9am sharp! alright! :laugh:
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
-
For (my extremely slowly) upcoming take over the world project I created a new (C#) serializer! Why? Well this take over the world project, without disclosing too much, is a document editor. I need to save those documents. Writing a Save() and Read() method is quite cumbersome... Serializer, XML Serializer and DataContractSerializer don't make the cut for obvious reason. JsonSerializer is too bloated (think a list of 1000 points, endlessly repeating "x =" or "{" or "}") plus it doesn't work very well with property of type object or List Enter my serializer, it's version tolerant, strongly typed, have some kind of header with all type info, then it's a stream of value and I think it's output quite a compact stream of data. And it works with PCL (i.e. desktop, UWP, IOS, Android, Linux with .NET core) And it just finally worked! this morning at 8:35AM! ;P (well, not completely.. I just tweaked a test which then failed.. apparently there is still a known bug...) Well all of that to say... I might make a CodeProject article if there is some interest for it! ;) As a side note, Not really a GitHub project since it's smack in the middle of my utility library and removing it is .. tedious since I shared plenty of reflection extension method....
All in one Menu-Ribbon Bar DirectX for WinRT/C# since 2013! Taking over the world since 1371!
Just curious, why not something like the Avro serializer (Serializing data with the Microsoft .NET Library for Avro[^]) or Protocol Buffers (https://developers.google.com/protocol-buffers/docs/csharptutorial[^])? Avro supports schemas and versioning, and both produce a fairly compact output and are cross platform compatible.