I thought .NET was supposed to make things easier, if anything, than unmanaged code.
-
I know this is a very special case but still i ran headlong into it. The easiest way to implement a B+ tree on disk is using a memory mapped file. I think this is what SQL Server does, but don't quote me. However, the only way you can access memory mapped files in C# is through .NET interop which makes it useless. Because one of the points of a memory mapped file is that you can do memory allocations that are backed by disk. There's no way in hell .NET can give you that in its current incarnation, even if one were to write a custom host, because of the way a GC system works. What I'd like var foo = new int[1000000]; // backed by disk, paged automatically What I'd have to do. somepointer = VirtualAlloc(...) Write(somepointer, data) etc etc basically it works like file i/o which defeats essentially the whole purpose. =(
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
That Ezra Taft Benson, while true, is one of the dumbest remarks I've seen, and is used by some groups to put down government. The point of government is to take your money and do what's best for the people as a whole. You can argue that isn't what happens in some cases, but that misses the point. Government generally is the only entity that takes risks when the reward isn't obvious. That where just about all of our technology comes from because industry won't invest without the promise of fairly quick rewards.
Dewey wrote:
The point of government is to take your money and do what's best for the people as a whole.
That's certainly not the point of the U.S. Federal gov't. The constitution makes it very clear that the Federal gov't is supposed to protect the people from foreign enemies, and not do much else that gets in the way of individual freedom and liberty. Of course certain forces have corrupted this to great extent.
The difficult we do right away... ...the impossible takes slightly longer.
-
I know this is a very special case but still i ran headlong into it. The easiest way to implement a B+ tree on disk is using a memory mapped file. I think this is what SQL Server does, but don't quote me. However, the only way you can access memory mapped files in C# is through .NET interop which makes it useless. Because one of the points of a memory mapped file is that you can do memory allocations that are backed by disk. There's no way in hell .NET can give you that in its current incarnation, even if one were to write a custom host, because of the way a GC system works. What I'd like var foo = new int[1000000]; // backed by disk, paged automatically What I'd have to do. somepointer = VirtualAlloc(...) Write(somepointer, data) etc etc basically it works like file i/o which defeats essentially the whole purpose. =(
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
Ok, This is the Lounge. THIS post here is purely programmer speak so ... take it to a forum. Pick one.
my posts of this nature were already weighed in on by Chris. they've been ruled okay for the lounge, so if you don't like them skip them or block me.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
i know that, but it's not efficient and doesn't solve the basic issue, which is persistent, rapidly searchable storage.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
If you want something to be truly rapid, you'd perhaps need to look at a native programming language like C++. Yes, .NET is supposed to make things easier, but not necessarily faster executing.
-
If you want something to be truly rapid, you'd perhaps need to look at a native programming language like C++. Yes, .NET is supposed to make things easier, but not necessarily faster executing.
I'm not looking for something that's bit level optimized. I'm fine with looking at complexity as for example O(log N), and that's fine, without worrying about the .NET overhead on top of the base functionality. Nah, mem mapped files, if i could use them like they were intended but under .NET, would make my programming task easier. Not necessarily make the program faster.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
I'm not looking for something that's bit level optimized. I'm fine with looking at complexity as for example O(log N), and that's fine, without worrying about the .NET overhead on top of the base functionality. Nah, mem mapped files, if i could use them like they were intended but under .NET, would make my programming task easier. Not necessarily make the program faster.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
honey the codewitch wrote:
I'm fine with looking at complexity as for example O(log N), and that's fine, without worrying about the .NET overhead on top of the base functionality.
Doing a full u-turn, are we? This was your response to another person on this thread:
honey the codewitch wrote:
but it's not efficient and doesn't solve the basic issue, which is persistent, rapidly searchable storage.
-
Yes, though it was added to .NET after my initial attempt at using mem mapped files from C#. Besides all that is is a wrapper like the one i had written years ago. It doesn't change the basic problem which is: var foo = new int[1000000]; //backed by disk, paged automatically, in C/C++ it's mainly because you can't use pointers in C#, and even if you use unsafe, you cannot pin objects to specific addresses in memory
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
honey the codewitch wrote:
it's mainly because you can't use pointers in C#, and even if you use unsafe, you cannot pin objects to specific addresses in memory
Yes, and that's by design.
-
I know this is a very special case but still i ran headlong into it. The easiest way to implement a B+ tree on disk is using a memory mapped file. I think this is what SQL Server does, but don't quote me. However, the only way you can access memory mapped files in C# is through .NET interop which makes it useless. Because one of the points of a memory mapped file is that you can do memory allocations that are backed by disk. There's no way in hell .NET can give you that in its current incarnation, even if one were to write a custom host, because of the way a GC system works. What I'd like var foo = new int[1000000]; // backed by disk, paged automatically What I'd have to do. somepointer = VirtualAlloc(...) Write(somepointer, data) etc etc basically it works like file i/o which defeats essentially the whole purpose. =(
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
Environments designated to making simple things simple tend to make complicated things more complicated. Or maybe not, in the case of .NET, there's the unsafe keyword which allows you to work on raw pointers. However, that indeed doesn't help much with disk stuff. See it this way, C# is the polar opposite of C here. In C, complex things and simple things work about the same way at the cost of simple things being ridiculously complicated. I remember a project of mine where I combined the strengths of two worlds, I did my low-level in C (technically C++, but I had raw pointers at work), exposed a C interface and did my business logic in C#. Such a mixed design is of course too easy to get wrong, but I think I've managed to make it make sense.
-
Environments designated to making simple things simple tend to make complicated things more complicated. Or maybe not, in the case of .NET, there's the unsafe keyword which allows you to work on raw pointers. However, that indeed doesn't help much with disk stuff. See it this way, C# is the polar opposite of C here. In C, complex things and simple things work about the same way at the cost of simple things being ridiculously complicated. I remember a project of mine where I combined the strengths of two worlds, I did my low-level in C (technically C++, but I had raw pointers at work), exposed a C interface and did my business logic in C#. Such a mixed design is of course too easy to get wrong, but I think I've managed to make it make sense.
I completely agree, and I've done similar. It's just that here, it's not really my goal to make something tied to the windows platform specifically, and pretty much no matter the route i take as soon as I'm relying on that mixed managed and unmanaged code I'm also inherently tying it to a platform. Sure I can cross compile the unmanaged code, but even then, it's one package per platform - not what i want.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
honey the codewitch wrote:
it's mainly because you can't use pointers in C#, and even if you use unsafe, you cannot pin objects to specific addresses in memory
Yes, and that's by design.
I'm aware of that. I worked on the Visual Studio team at microsoft. I know a little about .NET
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
honey the codewitch wrote:
I'm fine with looking at complexity as for example O(log N), and that's fine, without worrying about the .NET overhead on top of the base functionality.
Doing a full u-turn, are we? This was your response to another person on this thread:
honey the codewitch wrote:
but it's not efficient and doesn't solve the basic issue, which is persistent, rapidly searchable storage.
No I'm not, because in this case, there's an order of magnitude of difference in performance between using the windows memory paging system, and seeking around a disk file all the time. It changes the complexity. I'm not concerned about 10% improvements.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
I completely agree, and I've done similar. It's just that here, it's not really my goal to make something tied to the windows platform specifically, and pretty much no matter the route i take as soon as I'm relying on that mixed managed and unmanaged code I'm also inherently tying it to a platform. Sure I can cross compile the unmanaged code, but even then, it's one package per platform - not what i want.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
Don't quote me, but I am rather sure that you can't do that in a platform-agnostic manner. Going with P/Invoke or a low-level library in C will require platform ties as the C API doesn't know of the concept you need (hey, the C standard doesn't even have a concept of alignment). From what I know, the C++ standard library doesn't have this concept either and the moment your standard libraries don't provide something, you have to work with IFDEFs for platforms. Be it with .NET, C++ or any other language.
-
Don't quote me, but I am rather sure that you can't do that in a platform-agnostic manner. Going with P/Invoke or a low-level library in C will require platform ties as the C API doesn't know of the concept you need (hey, the C standard doesn't even have a concept of alignment). From what I know, the C++ standard library doesn't have this concept either and the moment your standard libraries don't provide something, you have to work with IFDEFs for platforms. Be it with .NET, C++ or any other language.
yeah but i think you can do p/invoke in mono to access ELF binaries (or whatever they'll called these days), etc. worst case I'd use #if in C# to plug in the right p/invoke attributes for the platform So like i said, it might be that I could cross compile the unmanaged bits, but in any case it doesn't give me what i want here.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
yeah but i think you can do p/invoke in mono to access ELF binaries (or whatever they'll called these days), etc. worst case I'd use #if in C# to plug in the right p/invoke attributes for the platform So like i said, it might be that I could cross compile the unmanaged bits, but in any case it doesn't give me what i want here.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
As I was saying, I am darn sure that it's not realistic to get this rather special use case of yours in a platform-agnostic manner. You don't need to cross-compile though. Of course can you P/Invoke in Mono and Mono has it's own share of IFDEFs to tell platforms apart. So compile ONE executable to run on Mono (or .NET Core which also has ways to tell platforms apart) but with your NativeMethods class being full of IFDEFs.
-
As I was saying, I am darn sure that it's not realistic to get this rather special use case of yours in a platform-agnostic manner. You don't need to cross-compile though. Of course can you P/Invoke in Mono and Mono has it's own share of IFDEFs to tell platforms apart. So compile ONE executable to run on Mono (or .NET Core which also has ways to tell platforms apart) but with your NativeMethods class being full of IFDEFs.
I'm pretty sure I'd need to cross compile the C code as Virtual memory management is completely different under windows than it is in linux for example. I don't think i can even map arbitrary files to process address space in linux but i could be wrong.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
I'm pretty sure I'd need to cross compile the C code as Virtual memory management is completely different under windows than it is in linux for example. I don't think i can even map arbitrary files to process address space in linux but i could be wrong.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
You would have to cross-compile if you go the route of compiling a separate library (in, let's say, C) for this stuff. But you can also go with P/Invoke which you "cross compile" via runtime checks. .NET Core supports this scenario, I haven't checked on Mono (yet). https://stackoverflow.com/questions/38790802/determine-operating-system-in-net-core
-
You would have to cross-compile if you go the route of compiling a separate library (in, let's say, C) for this stuff. But you can also go with P/Invoke which you "cross compile" via runtime checks. .NET Core supports this scenario, I haven't checked on Mono (yet). https://stackoverflow.com/questions/38790802/determine-operating-system-in-net-core
yeah in the general sense, but in this case, i'd need two binaries per platform (one managed, one unmanaged) probably. #ifs require compilation of course, in C# as well as C
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
yeah in the general sense, but in this case, i'd need two binaries per platform (one managed, one unmanaged) probably. #ifs require compilation of course, in C# as well as C
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
Why do you insist on an unmanaged binary? Why don't you want to do your unmanaged ABI calls with P/Invoke?
-
Why do you insist on an unmanaged binary? Why don't you want to do your unmanaged ABI calls with P/Invoke?
Because it's impossible to map a file to process address space in C#. You can only marshal calls to the vmem system which makes it exactly like reading and writing a file instead of reading and writing a pointer. Defeating the primary reason I'd use it. what I want (and can't have)
var foo = new int[100000]; // backed by disk automatically
and i don't want an unmanaged binary. I'm speaking in the hypothetical. I will not use vmem in this project because of .NET limitations.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
