I thought .NET was supposed to make things easier, if anything, than unmanaged code.
-
Exactly my point, keep it managed, save yourself the trouble of cross-compiling and furthermore, the distributional pain. Distributing different binaries for different platforms has been the norm for pretty much forever (in computing at least), but .NET Core/Mono provide means to do it better. Hey, remember when even DOS software had to have different binaries for different computer vendors despite them all running DOS? I surely don't miss those days.
I planned to keep it managed from the beginning. The only reason i discussed an unmanaged scenario is because memory mapped files basically don't work - at least how they were designed to be used - under .NET. I wanted people to understand the problem domain. I do not make mixed mode distributions anymore.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
I know this is a very special case but still i ran headlong into it. The easiest way to implement a B+ tree on disk is using a memory mapped file. I think this is what SQL Server does, but don't quote me. However, the only way you can access memory mapped files in C# is through .NET interop which makes it useless. Because one of the points of a memory mapped file is that you can do memory allocations that are backed by disk. There's no way in hell .NET can give you that in its current incarnation, even if one were to write a custom host, because of the way a GC system works. What I'd like var foo = new int[1000000]; // backed by disk, paged automatically What I'd have to do. somepointer = VirtualAlloc(...) Write(somepointer, data) etc etc basically it works like file i/o which defeats essentially the whole purpose. =(
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
[This StackOverflow question](https://stackoverflow.com/questions/31179076/how-to-get-an-intptr-to-access-the-view-of-a-memorymappedfile) appears to allow you to get a fixed pointer to a mapped view. Then you could use a `Span` object to provide an array representation over that memory range (use the [constructor that takes a `Void*` and `Int32`](https://docs.microsoft.com/en-us/dotnet/api/system.span-1.-ctor?view=netstandard-2.1#System\_Span\_1\_\_ctor\_System\_Void\_\_System\_Int32\_)) and then writes to the `Span` would be written to the mapped memory and thus to the mapped file?
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
-
[This StackOverflow question](https://stackoverflow.com/questions/31179076/how-to-get-an-intptr-to-access-the-view-of-a-memorymappedfile) appears to allow you to get a fixed pointer to a mapped view. Then you could use a `Span` object to provide an array representation over that memory range (use the [constructor that takes a `Void*` and `Int32`](https://docs.microsoft.com/en-us/dotnet/api/system.span-1.-ctor?view=netstandard-2.1#System\_Span\_1\_\_ctor\_System\_Void\_\_System\_Int32\_)) and then writes to the `Span` would be written to the mapped memory and thus to the mapped file?
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
Oooh now that's clever.I forgot about span! Hmmm. this might be doable. thanks for the link. Span is cool.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
I planned to keep it managed from the beginning. The only reason i discussed an unmanaged scenario is because memory mapped files basically don't work - at least how they were designed to be used - under .NET. I wanted people to understand the problem domain. I do not make mixed mode distributions anymore.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
I don't quite believe you that memory-mapped files don't work in .NET since .NET has a MemoryMappedFile class.
-
I don't quite believe you that memory-mapped files don't work in .NET since .NET has a MemoryMappedFile class.
I said they don't work how they are supposed to. How they were designed was to map files to a process address space so you could do pointer ops to read and write files. That is not doable under .NET. Sorry I've just explained this a lot. If you've used memory mapped files in unmanaged code then you know what i'm talking about. Otherwise you might never. I don't know. You cannot do var foo = new int[100000]; //backed by file. That's how it's supposed to work. It can't under .NET
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
Oooh now that's clever.I forgot about span! Hmmm. this might be doable. thanks for the link. Span is cool.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
No problem... Inspiration came from using [std::slice](https://doc.rust-lang.org/std/slice/fn.from\_raw\_parts.html) in Rust and [gsl::span](http://codexpert.ro/blog/2016/03/07/guidelines-support-library-review-spant/) in C++ - both create a non-owning array-like wrapper over a chunk of memory...
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
-
No problem... Inspiration came from using [std::slice](https://doc.rust-lang.org/std/slice/fn.from\_raw\_parts.html) in Rust and [gsl::span](http://codexpert.ro/blog/2016/03/07/guidelines-support-library-review-spant/) in C++ - both create a non-owning array-like wrapper over a chunk of memory...
Java, Basic, who cares - it's all a bunch of tree-hugging hippy cr*p
yeah. My professional dev years are behind me and so the newer .NET stuff I'm still picking up. I even stopped coding for years. But I remember span, now that you mention it of course. I thought it was one of the coolest new features of .NET =) Now if i can just remember it all the time.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
Yes, though it was added to .NET after my initial attempt at using mem mapped files from C#. Besides all that is is a wrapper like the one i had written years ago. It doesn't change the basic problem which is: var foo = new int[1000000]; //backed by disk, paged automatically, in C/C++ it's mainly because you can't use pointers in C#, and even if you use unsafe, you cannot pin objects to specific addresses in memory
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
If you want to do this in the dotNet framework and have it cross platform you're going to want to use one of the newer frameworks anyway as the older ones are not generally cross platform. I don't see the problem here.
a newer framework doesn't change the landscape though. I'm using the latest stuff anyway.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
C++/CLI
#SupportHeForShe Government can give you nothing but what it takes from somebody else. A government big enough to give you everything you want is big enough to take everything you've got, including your freedom.-Ezra Taft Benson You must accept 1 of 2 basic premises: Either we are alone in the universe or we are not alone. Either way, the implications are staggering!-Wernher von Braun
+1 for this suggestion. Performance is much, much better than using the managed classes for memory mapped files. (I remember we clocked this being around 5x faster). We used this approach to store multimedia fingerprint data. Here is a small snippet of code showing how we initialized the memory mapped file using C++/CLI - The data is accessible through the _pData (UInt32*) member.
Int32 StationHashStorage::Open() {
msclr::lock lock(_syncRoot);
if( _isOpen )
return 0;
String^ fileName = GetFullFileName();_szInBytes = ComputeFileSizeInBytes(fileName);
String^ mapExtension = GetFileExtension();
String^ mapName = String::Format("{0}{1}_{2}", _stationId, _date.ToString("yyyyMMdd"), mapExtension);marshal_context context;
LPCTSTR pMapName = context.marshal_as(mapName);{
msclr::lock lock( _openLock );
// Try to see if another storage instance has requested the same memory-mapped file and share it
_hMapping = OpenFileMapping(FILE_MAP_READ | FILE_MAP_WRITE, FALSE, pMapName);
if( !_hMapping ) {
// This is the first instance acquiring the file
LPCTSTR pFileName = context.marshal_as(fileName);
// Try to open the existing file, or create new one if not exists
_hFile = CreateFile(pFileName,
GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ,
NULL,
OPEN_ALWAYS,
FILE_ATTRIBUTE_NORMAL,
NULL);
if( !_hFile )
throw gcnew IOException(String::Format(Strings::CreateFileFailed, GetLastError(), _stationId));
_hMapping = CreateFileMapping(_hFile,
NULL,
PAGE_READWRITE | SEC_COMMIT,
0,
_szInBytes,
pMapName);
if( !_hMapping )
throw gcnew IOException(String::Format(Strings::CreateMappingFailed, GetLastError(), _stationId));
_usingSharedFile = false;
} else {
_usingSharedFile = true;
}
}_pData = (UInt32*)::MapViewOfFile(_hMapping, FILE_MAP_READ | FILE_MAP_WRITE, 0, 0, 0);
if( !_pData )
throw gcnew IOE -
I know this is a very special case but still i ran headlong into it. The easiest way to implement a B+ tree on disk is using a memory mapped file. I think this is what SQL Server does, but don't quote me. However, the only way you can access memory mapped files in C# is through .NET interop which makes it useless. Because one of the points of a memory mapped file is that you can do memory allocations that are backed by disk. There's no way in hell .NET can give you that in its current incarnation, even if one were to write a custom host, because of the way a GC system works. What I'd like var foo = new int[1000000]; // backed by disk, paged automatically What I'd have to do. somepointer = VirtualAlloc(...) Write(somepointer, data) etc etc basically it works like file i/o which defeats essentially the whole purpose. =(
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
I believe this is the type of library you are looking for... [C# BPlusTree](http://csharptest.net/projects/bplustree/)
Steve Naidamast Sr. Software Engineer Black Falcon Software, Inc. blackfalconsoftware@outlook.com
-
I believe this is the type of library you are looking for... [C# BPlusTree](http://csharptest.net/projects/bplustree/)
Steve Naidamast Sr. Software Engineer Black Falcon Software, Inc. blackfalconsoftware@outlook.com
I've seen that. This is a learning exercise for me, so I'm not looking for one that's already made.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
my posts of this nature were already weighed in on by Chris. they've been ruled okay for the lounge, so if you don't like them skip them or block me.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
honey the codewitch wrote:
my posts of this nature were already weighed in on by Chris
Careful. I said posts that talk about general programming stuff like "here's what I'm doing" are fine. Posts that are of a technical nature that are better suited to one of the dedicated programming forums should be posted in the appropriate forum in order to allow members to discuss programming topics. Yeah it's a grey area, but follow the spirit.
cheers Chris Maunder
-
honey the codewitch wrote:
my posts of this nature were already weighed in on by Chris
Careful. I said posts that talk about general programming stuff like "here's what I'm doing" are fine. Posts that are of a technical nature that are better suited to one of the dedicated programming forums should be posted in the appropriate forum in order to allow members to discuss programming topics. Yeah it's a grey area, but follow the spirit.
cheers Chris Maunder
I hear, you and I'm trying. My brain is all tech whenever i'm on a coding jag, so i eat sleep and breathe it.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
I hear, you and I'm trying. My brain is all tech whenever i'm on a coding jag, so i eat sleep and breathe it.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
You ever get to the point where you try and have conversations with non-coders and you get internal parsing errors? I guess the question could be "do you ever not get to the point..."
cheers Chris Maunder
-
You ever get to the point where you try and have conversations with non-coders and you get internal parsing errors? I guess the question could be "do you ever not get to the point..."
cheers Chris Maunder
yes i do. my solution thus far has been while i'm coding to limit my exposure to non-coders. :laugh: But i wind up here a lot lately while in the middle of it. Mainly because i look here for inspiration and a break from the actual writing of code. sometimes to vent about chicanery in MS products and the like too. Still, I can't turn my mind away from it entirely when i'm in the thick of it. I'm like a dog with a bone.
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
I know this is a very special case but still i ran headlong into it. The easiest way to implement a B+ tree on disk is using a memory mapped file. I think this is what SQL Server does, but don't quote me. However, the only way you can access memory mapped files in C# is through .NET interop which makes it useless. Because one of the points of a memory mapped file is that you can do memory allocations that are backed by disk. There's no way in hell .NET can give you that in its current incarnation, even if one were to write a custom host, because of the way a GC system works. What I'd like var foo = new int[1000000]; // backed by disk, paged automatically What I'd have to do. somepointer = VirtualAlloc(...) Write(somepointer, data) etc etc basically it works like file i/o which defeats essentially the whole purpose. =(
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
Use unsafe code with pointers just as you would in C and you can do the same thing as you could in C.
Nope, because you can't pin an object to a specific address in memory. Ergo I can't back a .NET instance with a memory mapped file. Memory mapped files work by mapping a portion of the process address space to your application. So you can read and write the file doing pointer ops. However, in .NET you can only get pointers to the GC heap by pinning objects. So I cannot do
var arr = new int[100000000];// backed by memory mapped file
in any variation. Even if I were to pin the array to get the pointer, the pointer wouldn't be part of that mapped address space. At best, I have to *copy* the entire object into that mapped region, which absolutely defeats the whole point of why I'd use mem mapped I/O here in the first place. There's a half measure using Span, but it only works in Standard and Core, not the DNF
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
-
Nope, because you can't pin an object to a specific address in memory. Ergo I can't back a .NET instance with a memory mapped file. Memory mapped files work by mapping a portion of the process address space to your application. So you can read and write the file doing pointer ops. However, in .NET you can only get pointers to the GC heap by pinning objects. So I cannot do
var arr = new int[100000000];// backed by memory mapped file
in any variation. Even if I were to pin the array to get the pointer, the pointer wouldn't be part of that mapped address space. At best, I have to *copy* the entire object into that mapped region, which absolutely defeats the whole point of why I'd use mem mapped I/O here in the first place. There's a half measure using Span, but it only works in Standard and Core, not the DNF
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.
This is what "Span" is for. Return that instead of an array and your API will be "managed" in the sense to an external user. Or you can use "GCHandle.Alloc" to pin a managed array in memory. GCHandle.Alloc Method (System.Runtime.InteropServices) | Microsoft Docs[^] Using Span is a better idea though.
-
This is what "Span" is for. Return that instead of an array and your API will be "managed" in the sense to an external user. Or you can use "GCHandle.Alloc" to pin a managed array in memory. GCHandle.Alloc Method (System.Runtime.InteropServices) | Microsoft Docs[^] Using Span is a better idea though.
Another commenter pointed out Span, which I had forgotten about. It's a half measure** but gets me closer, however, it does not work in the DNF, only in Core and Standard i think. GCHandle.Alloc will not let me pin to a specific address. In order for .NET to do anything beyond span it would have to allow you to create unmanaged heaps to allocate objects on which the collector wouldn't collect. It's designed such that it can't work, and I accept that. It would just be nice if it had some kind of feature like the above to enable it. ** I don't actually want to store an array, i was using that as an example. My datastructure is complex and nested, which is precisely why i'd like to use vmem
When I was growin' up, I was the smartest kid I knew. Maybe that was just because I didn't know that many kids. All I know is now I feel the opposite.