Appending data to a aes encrypted file?
-
Hello good people, I was wondering if it is possible to append data to an already encrypted file? I tried a few projects the past few days but always when i append data to the file, the second line and all lines after that contain mangled data. Using google to find out if it is possible or not did not give me any desirable results, so i wanted to ask in here if someone knows? Am i trying to do something that is not possible? Do i need to take a different approach, like first decrypting all data into a temporary file and then re-encrypt the whole lot? Any advise would be greatly appreciated, after three days of trying i am hoping someone could enlighten me. :laugh:
-
Hello good people, I was wondering if it is possible to append data to an already encrypted file? I tried a few projects the past few days but always when i append data to the file, the second line and all lines after that contain mangled data. Using google to find out if it is possible or not did not give me any desirable results, so i wanted to ask in here if someone knows? Am i trying to do something that is not possible? Do i need to take a different approach, like first decrypting all data into a temporary file and then re-encrypt the whole lot? Any advise would be greatly appreciated, after three days of trying i am hoping someone could enlighten me. :laugh:
Although it is possible, it would take a considerable amount of effort to append data to an already encrypted file - in effect running the encryption again to get the encrypter to the correct sate for the insert, including removing any padded final block padding before you append your data. The effort is the equivalent of re-writing the encryption routines. It would be considerably easier and a lot quicker to decrypt, append, and re-encrypt again. Do NOT use a temporary file, use a memory stream instead as this will not be left on the HDD (unless it is paged off to free memory). A temporary files' contents will be left on the HDD even if the temporary file is deleted, until that disk space is re-used by some other application.
You should never use standby on an elephant. It always crashes when you lift the ears. - Mark Wallace C/C++ (I dont see a huge difference between them, and the 'benefits' of C++ are questionable, who needs inheritance when you have copy and paste) - fat_boy
-
Hello good people, I was wondering if it is possible to append data to an already encrypted file? I tried a few projects the past few days but always when i append data to the file, the second line and all lines after that contain mangled data. Using google to find out if it is possible or not did not give me any desirable results, so i wanted to ask in here if someone knows? Am i trying to do something that is not possible? Do i need to take a different approach, like first decrypting all data into a temporary file and then re-encrypt the whole lot? Any advise would be greatly appreciated, after three days of trying i am hoping someone could enlighten me. :laugh:
Does it even matter? You can't hide anything from the user anyway - if your application has the key and the user has the application then the user has the key, QED. Anyway, in general it shouldn't be possible. AES is a block algorithm, so there will be padding, and you can't remove the padding without decrypting, because the padding is encrypted. Therefore at the very least, you will have to decrypt the last block. For the rest I am not sure, but I think I remember something about mixing the key with the data? If that is what it does then you can't just "run the key stream* X bytes where X is the length of the file" as you might otherwise do, since in order to do that you'd need to have the decrypted data an you don't have it. Unless of course you do have it, but you seem to want to avoid that. (why? it's not like you're keeping a secret from your users anyway, as proven above. decrypting to ram isn't going to make the situation any worse, although it may make the process slower) * incorrect terminology note: as always with posts about "hard" subjects, I can't guarantee the correctness of the contents of this post - use at your own risk.
-
Although it is possible, it would take a considerable amount of effort to append data to an already encrypted file - in effect running the encryption again to get the encrypter to the correct sate for the insert, including removing any padded final block padding before you append your data. The effort is the equivalent of re-writing the encryption routines. It would be considerably easier and a lot quicker to decrypt, append, and re-encrypt again. Do NOT use a temporary file, use a memory stream instead as this will not be left on the HDD (unless it is paged off to free memory). A temporary files' contents will be left on the HDD even if the temporary file is deleted, until that disk space is re-used by some other application.
You should never use standby on an elephant. It always crashes when you lift the ears. - Mark Wallace C/C++ (I dont see a huge difference between them, and the 'benefits' of C++ are questionable, who needs inheritance when you have copy and paste) - fat_boy
Thank you for your swift answer OriginalGriff. Let's see if i understand this correctly then, i will try and use your suggested method. However using MemoryStream to decrypt, append and then rewrite the whole data into the same file wouldn't that overwrite that same file? Using a memory stream to let's say decrypt a big file into memory, let's say 200MB wouldn't that affect system speed?
-
Hello good people, I was wondering if it is possible to append data to an already encrypted file? I tried a few projects the past few days but always when i append data to the file, the second line and all lines after that contain mangled data. Using google to find out if it is possible or not did not give me any desirable results, so i wanted to ask in here if someone knows? Am i trying to do something that is not possible? Do i need to take a different approach, like first decrypting all data into a temporary file and then re-encrypt the whole lot? Any advise would be greatly appreciated, after three days of trying i am hoping someone could enlighten me. :laugh:
Well it seems i need to do a lot more reading then, any good suggestions on books or online material that touches this subject? It's critical to my application to store the data encrypted if possible, only the information supplied by the user should be encrypted in one file. I would like to thank you both about shedding some light on this topic.
-
Hello good people, I was wondering if it is possible to append data to an already encrypted file? I tried a few projects the past few days but always when i append data to the file, the second line and all lines after that contain mangled data. Using google to find out if it is possible or not did not give me any desirable results, so i wanted to ask in here if someone knows? Am i trying to do something that is not possible? Do i need to take a different approach, like first decrypting all data into a temporary file and then re-encrypt the whole lot? Any advise would be greatly appreciated, after three days of trying i am hoping someone could enlighten me. :laugh:
It takes a bit of work, but you could code an interface to write a 'footer' that contains indexes of encrypted blocks within the file. So for instance, - you would write 1024 encrypted bytes - then write a footer that says the index 0 file ends at 1024. - when you want to append, read the footer to memory. - shorten the file to that last index (erasing the footer). - write your new encrypted block. - write the new index (1) to footer in memory. - write the new footer to the end of the file. The downside (potentially) is that if you want to remove a file that is not at the last index, you essentially have to rewrite the entire file.
-
Thank you for your swift answer OriginalGriff. Let's see if i understand this correctly then, i will try and use your suggested method. However using MemoryStream to decrypt, append and then rewrite the whole data into the same file wouldn't that overwrite that same file? Using a memory stream to let's say decrypt a big file into memory, let's say 200MB wouldn't that affect system speed?
Balletje wrote:
Using a memory stream to let's say decrypt a big file into memory, let's say 200MB wouldn't that affect system speed?
Yes. It may speed it up... It's one of "those answers" which depends on too many variables: Does your system cache HDD writes? Does the HDD hardware cache writes, and is it big enough? Think about it: If the stream stays in memory, it will be faster than writing to HDD (unless the HDD data also stays in memory). If it gets paged out, then it will be at about the same speed as the temporary file solution, except any other activity in the PC will slow down. For me, the security risk of leaving unencrypted data just lying around is too high - it defeats the purpose of encryption in the first place. I always decrypt to memory - I may have no control over the page table, nor how many fragments of unencrypted data are in memory, but at least I can be as confident as possible that if you turn the power off, it will be a non-trivial task to access the unencrypted data!
You should never use standby on an elephant. It always crashes when you lift the ears. - Mark Wallace C/C++ (I dont see a huge difference between them, and the 'benefits' of C++ are questionable, who needs inheritance when you have copy and paste) - fat_boy
-
Balletje wrote:
Using a memory stream to let's say decrypt a big file into memory, let's say 200MB wouldn't that affect system speed?
Yes. It may speed it up... It's one of "those answers" which depends on too many variables: Does your system cache HDD writes? Does the HDD hardware cache writes, and is it big enough? Think about it: If the stream stays in memory, it will be faster than writing to HDD (unless the HDD data also stays in memory). If it gets paged out, then it will be at about the same speed as the temporary file solution, except any other activity in the PC will slow down. For me, the security risk of leaving unencrypted data just lying around is too high - it defeats the purpose of encryption in the first place. I always decrypt to memory - I may have no control over the page table, nor how many fragments of unencrypted data are in memory, but at least I can be as confident as possible that if you turn the power off, it will be a non-trivial task to access the unencrypted data!
You should never use standby on an elephant. It always crashes when you lift the ears. - Mark Wallace C/C++ (I dont see a huge difference between them, and the 'benefits' of C++ are questionable, who needs inheritance when you have copy and paste) - fat_boy
:laugh: Ahhhh yes how stupid of me....... I will go and try to construct a method that will decrypt the file into memory, append last data added to the end and re-encrypt the whole deal back. Still i think i need to go and try to find some good lecture about this matter as i am having a bit of trouble understanding how it works. Though people's input here has been great and i greatly appreciate it. Thank you for explaining! :thumbsup:
-
Well it seems i need to do a lot more reading then, any good suggestions on books or online material that touches this subject? It's critical to my application to store the data encrypted if possible, only the information supplied by the user should be encrypted in one file. I would like to thank you both about shedding some light on this topic.
-
Btw, if you're just hiding the data from other users, wouldn't it be a lot easier to just use build-in NTFS encryption?
Well the application stores information about servers, their locations and needed passwords. That was basically the whole reason why i wanted that file encrypted because *should* it fall in the wrong hands they will at least have some trouble getting the information from it. But i am quite new to the whole encryption stuff, so i did not really know which way to take.
-
Well the application stores information about servers, their locations and needed passwords. That was basically the whole reason why i wanted that file encrypted because *should* it fall in the wrong hands they will at least have some trouble getting the information from it. But i am quite new to the whole encryption stuff, so i did not really know which way to take.
I see, well the build-in encryption is transparent so you don't have to do anything special - just set a flag that you want the file to be encrypted. The encryption is linked to the password of the current user, which is not the hardest thing to find out these days, but then that's probably true of the key used by your application as well..
