Styjun

I am trying AES decrypt operation using openssl functions from C code, but it is failing.

Using openssl command line tools I can decrypt the blob successfully.

openssl enc -d -p -aes-256-cbc -md md5 -in encrypted_file -out clear_file -pass file:./key_file -v

The above command works fine.

But when I use openssl C functions to do the same it fails. The failure seems to be related to wrong key and iv derived from passwd and salt.

unsigned char key[32];
unsigned char iv[16];
EVP_BytesToKey(EVP_aes_256_cbc(), EVP_md5(), salt, key_file_buf, key_size, 1, key, iv);

[key_file_buf is an unsigned char buffer read from key_file.]

Hexdump of salt and key_file_buf matches with what is used in command line. Size is also correct.(45 bytes in my case.)

What could be going wrong with EVP_BytesToKey() usage to return wrong key and iv?

I've tried experimenting with iter count values, but none seems to generate the working key and iv. I assume the command line default iter count is 1 anyway.

Also confirmed, if I overwrite what is returned from EVP_BytesToKey() and hard code unsigned char arrays with the working key and iv shown from command line rest of my code works fine and decrypts correctly.

For info, this is how rest of the code looks like (copied from different sources, examples on the web)

EVP_CIPHER_CTX_new();
if(ctx == NULL) 
 printf("Error with EVP_CIPHER_CTX_new.n");
 return;
 

if(1 != EVP_DecryptInit_ex(ctx, EVP_aes_256_cbc(), NULL, key, iv)) 
 printf("Error initialising decrypted data.n");
 return;
 

if(1 != EVP_DecryptUpdate(ctx, clear_data, (int *)&interm_len, &enc_data[salt_size], enc_size)) 
 printf("Error decrypting data.n");
 return;
 

*clear_size = interm_len;

if(1 != EVP_DecryptFinal_ex(ctx, clear_data + interm_len, (int *)&interm_len)) 
 printf("Error decrypting data.n");
 return;
 
*clear_size += interm_len;

EVP_CIPHER_CTX_free(ctx);

Can anyone please help?

Figured out finally!
Should have been

EVP_BytesToKey(EVP_aes_256_cbc(), EVP_md5(), salt, key_file_buf, (key_size-1), 1, key, iv);

As explained in openssl documentation

file:pathname
The first line of pathname is the password. If the same pathname argument is supplied to -passin and -passout arguments then the first line will be used for the input password and the next line for the output password.

My passwd file ends in newline 0x0A. So I removed that from the buf for EVP_BytesToKey() and it returns correct key & iv and decrypts fine now.