Just one well placed virtual function saves the day
-
I've been trying to speed up my JSON processing and the bottleneck was my I/O class. So I added a function to my I/O classes called skipToAny() which takes a string of characters and advances the input until it finds one of them. Depending on the kind of source you use (like memory mapped file sources) it's *fast* apparently strpbrk() is optimized for SSE and AVX instructions w/ some stdlib implementations. So using the builtin C function to scan over my memory mapped buffer increases my performance by a factor of almost 4. I'm in spitting distance of 600MB/s now. And I didn't even need to bit twiddle. Higher level optimizations > bit twiddling
Real programmers use butterflies
-
I've been trying to speed up my JSON processing and the bottleneck was my I/O class. So I added a function to my I/O classes called skipToAny() which takes a string of characters and advances the input until it finds one of them. Depending on the kind of source you use (like memory mapped file sources) it's *fast* apparently strpbrk() is optimized for SSE and AVX instructions w/ some stdlib implementations. So using the builtin C function to scan over my memory mapped buffer increases my performance by a factor of almost 4. I'm in spitting distance of 600MB/s now. And I didn't even need to bit twiddle. Higher level optimizations > bit twiddling
Real programmers use butterflies
This is a very specific topic but oddly I received an email from a colleague today about a library called simdjson which gives a significant performance improvement over other c++ json libraries. simdjson claim 'Standalone UTF8 Validation' at 13/GB/s
-
This is a very specific topic but oddly I received an email from a colleague today about a library called simdjson which gives a significant performance improvement over other c++ json libraries. simdjson claim 'Standalone UTF8 Validation' at 13/GB/s
simdjson is cool, but it doesn't run everywhere. My JSON thing will even run on arduinos
Real programmers use butterflies
-
simdjson is cool, but it doesn't run everywhere. My JSON thing will even run on arduinos
Real programmers use butterflies
-
What are you doing on an Arduino that needs to process large Json files so quickly? Or is this just a case of seeing how many people can fit in a phone booth?
If you can't laugh at yourself - ask me and I will do it for you.
It's not fast on an 8 bit Arduino, it's just that it scales to workstations and servers. The advantage of it over ArduinoJSON is the same library also blazes on traditional computers, and it actually uses far less RAM than ArduinoJSON, which cannot process large online dumps from say, a mongoDB backed repository, like the one tbdb.com runs. ETA: Originally I wrote it to try out a novel way of parsing and running queries, but it turns out simdjson just recently invented something similar. =)
Real programmers use butterflies