Generate SHA256 Hash of every file on my computer.
-
If You Give A Mouse A Cookie* I'm on a path to do a weird thing: generate the SHA256 hash of every file on my computer (store filename and hash in db). To do that, I started thinking about how I might set multiple instances of the process to do a portion of the work so it'll be (overall) faster. I figure I could write every directory on my system to a sqlite db then let numerous processes grab a folder, get all files, independently of each other. That made me start wondering about how much data it would be to store every directory on my system in a sqlite database. Wrote A Quick C# Program I wrote a quick little C# program that: 1. user gives starting path 2. program iterates through every directory 3. writes each directory to the sqlite db. Fast Iteration, Slow Insert I (of course) discovered : 1. it's super fast to iterate over the directories. It can iterate over 239618 directories in my Linux user directory in a few seconds. 2. It's super slow to use EntityFramework to insert those dir names into the sqlite db. Super slow means it takes more than 10 minutes. Two Weird Parts (but maybe expected) So, instead of inserting the records into the db directly I write data to a file (yes,it's 239,618 lines long) Data looks like (pipe delimited):
239609|/home/fakepath/faker|2024-10-15 17:16:27
Weird 1 C# iterates all those directories and writes to file in 2-3 seconds Weird 2 (also kind of wonderful) sqlite imports that data (over 239 thousand rows) in less than 1 second on the SQLite command line: > .import allPaths.dat finfo Import command takes file name with data and the target table name (finfo) I'm figuring this will make a lot of people say, "yeah, I figured so". :-D *In that book, one thing leads to another. Give a mouse a cookie, he'll want some milk. To get the milk you'll have to...
-
If You Give A Mouse A Cookie* I'm on a path to do a weird thing: generate the SHA256 hash of every file on my computer (store filename and hash in db). To do that, I started thinking about how I might set multiple instances of the process to do a portion of the work so it'll be (overall) faster. I figure I could write every directory on my system to a sqlite db then let numerous processes grab a folder, get all files, independently of each other. That made me start wondering about how much data it would be to store every directory on my system in a sqlite database. Wrote A Quick C# Program I wrote a quick little C# program that: 1. user gives starting path 2. program iterates through every directory 3. writes each directory to the sqlite db. Fast Iteration, Slow Insert I (of course) discovered : 1. it's super fast to iterate over the directories. It can iterate over 239618 directories in my Linux user directory in a few seconds. 2. It's super slow to use EntityFramework to insert those dir names into the sqlite db. Super slow means it takes more than 10 minutes. Two Weird Parts (but maybe expected) So, instead of inserting the records into the db directly I write data to a file (yes,it's 239,618 lines long) Data looks like (pipe delimited):
239609|/home/fakepath/faker|2024-10-15 17:16:27
Weird 1 C# iterates all those directories and writes to file in 2-3 seconds Weird 2 (also kind of wonderful) sqlite imports that data (over 239 thousand rows) in less than 1 second on the SQLite command line: > .import allPaths.dat finfo Import command takes file name with data and the target table name (finfo) I'm figuring this will make a lot of people say, "yeah, I figured so". :-D *In that book, one thing leads to another. Give a mouse a cookie, he'll want some milk. To get the milk you'll have to...
raddevus wrote:
*In that book, one thing leads to another. Give a mouse a cookie, he'll want some milk. To get the milk you'll have to...
Which leads to "If you give a Pig a Pancake" and "If you give a Moose a Muffin"
I’ve given up trying to be calm. However, I am open to feeling slightly less agitated. I’m begging you for the benefit of everyone, don’t be STUPID.
-
If You Give A Mouse A Cookie* I'm on a path to do a weird thing: generate the SHA256 hash of every file on my computer (store filename and hash in db). To do that, I started thinking about how I might set multiple instances of the process to do a portion of the work so it'll be (overall) faster. I figure I could write every directory on my system to a sqlite db then let numerous processes grab a folder, get all files, independently of each other. That made me start wondering about how much data it would be to store every directory on my system in a sqlite database. Wrote A Quick C# Program I wrote a quick little C# program that: 1. user gives starting path 2. program iterates through every directory 3. writes each directory to the sqlite db. Fast Iteration, Slow Insert I (of course) discovered : 1. it's super fast to iterate over the directories. It can iterate over 239618 directories in my Linux user directory in a few seconds. 2. It's super slow to use EntityFramework to insert those dir names into the sqlite db. Super slow means it takes more than 10 minutes. Two Weird Parts (but maybe expected) So, instead of inserting the records into the db directly I write data to a file (yes,it's 239,618 lines long) Data looks like (pipe delimited):
239609|/home/fakepath/faker|2024-10-15 17:16:27
Weird 1 C# iterates all those directories and writes to file in 2-3 seconds Weird 2 (also kind of wonderful) sqlite imports that data (over 239 thousand rows) in less than 1 second on the SQLite command line: > .import allPaths.dat finfo Import command takes file name with data and the target table name (finfo) I'm figuring this will make a lot of people say, "yeah, I figured so". :-D *In that book, one thing leads to another. Give a mouse a cookie, he'll want some milk. To get the milk you'll have to...
raddevus wrote:
2. It's super slow to use EntityFramework to insert those dir names into the sqlite db. Super slow means it takes more than 10 minutes.
Take a look at my article[^] about sqlite multi-threading. Might give you some ideas how to speed up things.
Mircea (see my latest musings at neacsu.net)