Some time ago I've needed to make client for one client-server protocol, where one step included blowfish-cfb encryption/decryption of data. Server side was written in PHP and used mcrypt library for encryption (need to note - mcrypt died in 2003). On server side it looked like that:
$result = mcrypt_decrypt(MCRYPT_BLOWFISH, $key, $data, MCRYPT_MODE_CFB);
$result = mcrypt_encrypt(MCRYPT_BLOWFISH, $key, $data, MCRYPT_MODE_CFB);
and that was bad code, bcs there are no 5th parameter. It set indirectly to \0\0\0\0\0\0\0\0. and it worked only before PHP 5.6 - after it won't work. In documentation written: "Invalid key and iv sizes are no longer accepted. mcrypt_encrypt() will now throw a warning and return FALSE if the inputs are invalid. Previously keys and IVs were padded with '\0' bytes to the next valid size." but this is not a problem - it's just little sad moment.
I've used c++ and OpenSSL - so, I've found description such functional in documentation (https://www.openssl.org/docs/crypto/blowfish.html):
void BF_cfb64_encrypt(const unsigned char *in, unsigned char *out,
long length, BF_KEY *schedule, unsigned char *ivec, int *num,
int enc);
"BF_cfb64_encrypt() is the CFB mode for Blowfish with 64 bit feedback."
BF_cfb64_encrypt was the only function in openssl, which matched what I searched - blowfish & cfb
I've made encryption/decryption function using OpenSSL and found a surprise - the same data, encrypted with the same key & same 'iv' after c++&openssl and after php&mcrypt was completely different. Well, not completely - only the first byte was the same. So, it was very strange: in both cases I had some classical algothitm - blowfish - which exists more than 20 years; and some standard mode - cfb. I've spent a lot of time (something like 3 days) for understanding - I've loaded php script into php interpretator, and opened this interpretator in ollydbg, and debugged debugged debugged .
Openssl and mcrypt - both of these libraries - it's like framework, where you can connect primitives, but with some difference:
- in mcrypt you have privitive of CFB and block cipher Blowfish - and they somehow connected during compiling & runtime - so, in sources you can't understand how it connected.
- in openssl you have BF_cfb64_encrypt, but ciphers can be connected into chains - and it happens also during compiling & runtime.
I wanted to write only this note, but during this research appeared 3 more notes.
And I've read a lot of discussions of this problem - and didn't find one simple answer. So, that's why I decided to write of this problem.
problem
When I encrypt data with blowfish in cfb mode on openssl - I can't decrypt on mcrypt correctly.
and vice verca:
When I encrypt data with mcrypt in cfb mode on openssl - I can't decrypt on openssl correctly.
It looked like these libraries incompatible, despite they implement the same.
This problem affects a lot of languages (I've seen bindings to: c/c++, perl, python, php, ruby, rust, nodejs, java, even haskell, lisp, lua, go, rust and some languages which I've not heard - nim, racket), because openssl & mcrypt - ones of the most famous crypto libraries, and any wide-spreaded programming language have bindings/wrappers/modules to these libraries. And if someone writes it's own implementation of blowfish-cfb - this person will likely check is it work correct, in comparison with one of these libraries. Such examples:
- C# bouncycastle library (behave as openssl)
- python crypto (behave as mcrypt by default)
Last commit to mcrypt project was in 2003 [link]. You shouldn't use it in your new projects.
to pass lyrics and just get solution - just scroll to "problem solve in a short".
illustration of a problem
php mcrypt
#--------------------------------------------------------------
<?php
$e = mcrypt_encrypt(MCRYPT_BLOWFISH, "mypass", "my_lovely_text", MCRYPT_MODE_CFB, "\0\0\0\0\0\0\0\0");
echo "$e";
?>
#--------------------------------------------------------------
3C 38 BA 2E-46 C4 2A 55-38 07 1F DE-89 E3
perl openssl
#--------------------------------------------------------------
use Crypt::OpenSSL::Blowfish::CFB64;
my $crypt = Crypt::OpenSSL::Blowfish::CFB64->new('mypass', pack( C8 => 0,0,0,0,0,0,0,0 ));
my $binary_data = $crypt->encrypt("my_lovely_text");
print $binary_data;
#--------------------------------------------------------------
3C 49 42 3A-0F 4F 65 50-67 29 74 36-31 37
C/C++ openssl:
//------------------------------------------------------------
#include <string.h>
#include <openssl/blowfish.h>
int main(int argc, char** argv)
{
const unsigned char plaintext[] = "my_lovely_text";
const unsigned char key[] = "mypass";
unsigned char out_buf[1024];
int counter = 0;
unsigned char iv[8];
BF_KEY keyStruct;
memset(iv, 0, 8);
BF_set_key(&keyStruct, strlen((const char*)key), key);
BF_cfb64_encrypt(
plaintext,
out_buf,
strlen((const char*)plaintext),
&keyStruct,
iv,
&counter,
BF_ENCRYPT
);
std::ofstream o("file.bf", std::ios::binary);
o.write((char*)out_buf, plaintext_len);
o.close();
return 0;
}
//------------------------------------------------------------
3c 49 42 3a 0f 4f 65 50 67 29 74 36 31 37
same_problems
I've found in the Internet several similar problems:
0) https://github.com/tugrul/node-mcrypt
The author writes:
"There is already OpenSSL extension bundled with Node.js but there are something wrong with some traditional encryption algorithms on OpenSSL"
Someone tired from this discrepancy and made his own wrapper on mcrypt.
This is not looks like answer for question 'why?'.
1) here are some Russian guy with such problem
http://phpclub.ru/talk/threads/delphi-php-mcrypt-blowfish.53690/
http://www.sql.ru/forum/544138/delphi-php-mcrypt-blowfish
Nodoby proposed good hypothesis.
2) here are also some Russian guy with such problem
http://ru-root.livejournal.com/2759003.html
and solution he found - use mcrypt instead of openssl :D
But he noticed that the 1st byte is the same. When I dealt with that - I didn't notice that.
And actually he was the most closest to the solving problem.
And in comments there was right answer, but I missed it when googled answer.
3) here are encrypt data with php and trying decrypt with nodejs, also check if everything is correct with openssl console tool (this theme I've mentioned here)
http://stackoverflow.com/questions/8895370/decrypting-mcrypt-encoded-text-with-node-js
so, everybody adviced:
- use some padding
- use iv
- use for decryption the same library which encrypted data
and nobody seen the reason - why it happens? :-/
lyrics
to make long story short - I will pass my adventures with olly debugger & just leave here source of
- openssl blowfish encryption in cfb mode, and
- generic cfb algorithm - from _mcrypt, where _mcrypt_block_encrypt will be blowfish block encryption.
// openssl blowfish cfb
// openssl-1.0.1j\crypto\bf\bf_cfb64.c
#define BF_ENCRYPT 1
#define BF_DECRYPT 0
void BF_cfb64_encrypt(const unsigned char *in, unsigned char *out, long length,
const BF_KEY *schedule, unsigned char *ivec, int *num, int encrypt)
{
register BF_LONG v0,v1,t;
register int n= *num;
register long l=length;
BF_LONG ti[2];
unsigned char *iv,c,cc;
iv=(unsigned char *)ivec;
if (encrypt)
{
while (l--)
{
if (n == 0)
{
n2l(iv,v0); ti[0]=v0;
n2l(iv,v1); ti[1]=v1;
BF_encrypt((BF_LONG *)ti,schedule);
iv=(unsigned char *)ivec;
t=ti[0]; l2n(t,iv);
t=ti[1]; l2n(t,iv);
iv=(unsigned char *)ivec;
}
c= *(in++)^iv[n];
*(out++)=c;
iv[n]=c;
n=(n+1)&0x07;
}
}
else
{
while (l--)
{
if (n == 0)
{
n2l(iv,v0); ti[0]=v0;
n2l(iv,v1); ti[1]=v1;
BF_encrypt((BF_LONG *)ti,schedule);
iv=(unsigned char *)ivec;
t=ti[0]; l2n(t,iv);
t=ti[1]; l2n(t,iv);
iv=(unsigned char *)ivec;
}
cc= *(in++);
c=iv[n];
iv[n]=cc;
*(out++)=c^cc;
n=(n+1)&0x07;
}
}
v0=v1=ti[0]=ti[1]=t=c=cc=0;
*num=n;
}
// mcrypt blowfish cfb
// mcrypt\modules\modes\cfb.c
int _mcrypt( CFB_BUFFER* buf, void *plaintext, int len, int blocksize, void* akey, void (*func)(void*,void*), void (*func2)(void*,void*))
{ /* plaintext is 1 byte (8bit cfb) */
char *plain = plaintext;
int i, j;
void (*_mcrypt_block_encrypt) (void *, void *);
_mcrypt_block_encrypt = func;
for (j = 0; j < len; j++) {
memcpy(buf->enc_s_register, buf->s_register, blocksize);
_mcrypt_block_encrypt(akey, buf->enc_s_register);
plain[j] ^= buf->enc_s_register[0];
/* Shift the register */
for (i = 0; i < (blocksize - 1); i++)
buf->s_register[i] = buf->s_register[i + 1];
buf->s_register[blocksize - 1] = plain[j];
}
return 0;
}
so, the problem is - it's two different algorithms!
mcrypt xor 8-bytes block and then move window to 1 byte.
openssl xor 8-bytes block and then move window to 8 byte.
and wikipedia says (http://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_Feedback_.28CFB.29)
"This way of proceeding is known as CFB-8 or CFB-1 (according to the size of the shifting)"
As I see in source code - it often calls cfb64 and cfb8 (instead of cfb-8 and cfb-1), but the point is correct - cfb can be different.
And now, looking in sources, it's clear answer for question - "why the first byte always the same?".
solving
I already had openssl - built, included, proven. So, I've just made my own implementation of cfb8 on c++. It's mcrypt-compatible algorithm (I've tested it, of course), which imports BF_encrypt from openssl library.
#include <openssl/blowfish.h>
#include <vector>
#define endian_swap32(x) ((x << 24) | (x >> 24) | ((x&0xff00)<<8) | ((x&0xff0000)>>8))
void BlowfishEncryptCFB8(
const std::vector<unsigned char> & in_buf,
const std::vector<unsigned char> & key,
const std::vector<unsigned char> & iv,
std::vector<unsigned char> & out_buf
)
{
out_buf = in_buf;
std::vector<unsigned char> internal_key(sizeof(BF_KEY));
BF_KEY *key_ = (BF_KEY *)internal_key.data();
BF_set_key(key_, key.size(), (const unsigned char*)key.data());
std::vector<unsigned char> enc_s_register(8), s_register;
s_register = iv;
for (size_t i = 0; i < in_buf.size(); i++)
{
enc_s_register = s_register;
__int32 * lp_enc_s_register = (__int32 *)enc_s_register.data();
*lp_enc_s_register = endian_swap32(*lp_enc_s_register);
*(lp_enc_s_register+1) = endian_swap32(*(lp_enc_s_register+1));
BF_encrypt((BF_LONG *) enc_s_register.data(), key_);
*lp_enc_s_register = endian_swap32(*lp_enc_s_register);
*(lp_enc_s_register+1) = endian_swap32(*(lp_enc_s_register+1));
out_buf[i] ^= enc_s_register[0];
for (size_t j = 0; j < 7; j++)
s_register[j] = s_register[j+1];
s_register[7] = out_buf[i];
}
}
void BlowfishEncryptCFB8(
const std::vector<unsigned char> & in_buf,
const std::vector<unsigned char> & key,
const std::vector<unsigned char> & iv,
std::vector<unsigned char> & out_buf
)
{
out_buf = in_buf;
std::vector<unsigned char> internal_key(sizeof(BF_KEY));
BF_KEY *key_ = (BF_KEY *)internal_key.data();
BF_set_key(key_, key.size(), (const unsigned char*)key.data());
std::vector<unsigned char> enc_s_register(8), s_register;
s_register = iv;
for (size_t i = 0; i < in_buf.size(); i++)
{
enc_s_register = s_register;
__int32 * lp_enc_s_register = (__int32 *)enc_s_register.data();
*lp_enc_s_register = endian_swap32(*lp_enc_s_register);
*(lp_enc_s_register + 1) = endian_swap32(*(lp_enc_s_register + 1));
BF_encrypt((BF_LONG *)enc_s_register.data(), key_);
*lp_enc_s_register = endian_swap32(*lp_enc_s_register);
*(lp_enc_s_register + 1) = endian_swap32(*(lp_enc_s_register + 1));
unsigned char saved_byte = out_buf[i];
out_buf[i] ^= enc_s_register[0];
for (size_t j = 0; j < 7; j++)
s_register[j] = s_register[j + 1];
s_register[7] = saved_byte;
}
}
problem solving in a short
- Realize that CFB mode - it's not some concrete action - it's common name for idea use previous ciphertext as key for next block.
- Realize that OpenSSL blowfish cfb - it's cfb64, and uses 64-bits feedback (that's why in sources this function called BF_cfb64_encrypt)
- Realize that mcrypt blowfish cfb - it's cfb8, and uses 8-bits feedback.
- determie which cfb mode do you have - for example, you can encode
- key: mypass
- text: my_lovely_text
- iv: 0
- blowfish in cfb mode:
- 3C 38 BA 2E-46 C4 2A 55-38 07 1F DE-89 E3 cfb-8 (mcrypt-like)
- 3C 49 42 3A-0F 4F 65 50-67 29 74 36-31 37 cfb-64 (openssl-like)
- if you are using something based on mcrypt and want openssl-compatibility:
- search something like 'ncfb' mode - it must be openssl-compatible cfb mode.
- else you will have simple choice:
- find implementation of mode which you need
- make your own implementation of cfb - block blowfish you should already have.
useful sources & links
After I've found answer by myself, I've found useful these links:
- http://ru-root.livejournal.com/2759003.html?thread=41331035#t41331035
- http://metaclass.livejournal.com/889791.html
- https://bugs.php.net/bug.php?id=51146
- http://stackoverflow.com/questions/4052594/incorrect-key-size-in-php-mcrypt-when-porting-cryptopp-aes-encryption-to-phps-m/4054017#4054017
- http://thefsb.tumblr.com/post/111035508040/porting-php-code-from-mcrypt-to-openssl
final notes
Aes has cfb8 and cfb1 modes - https://www.openssl.org/docs/apps/enc.html
Interesting - why blowfish doesn't have such choice, I wonder?
here are comparison openssl with mcrypt:
http://www.synet.sk/php/en/320-benchmarking-symmetric-cyphers-openssl-vs-mcrypt-in-php
Conclusions from this link: "OpenSSL offers significantly faster algorithms. This is the right choice for heavy traffic sites."
well, on blowfish-cfb it just makes in 8 time less work :-D
"The source code in CSV was last updated in October 2003. The only file changed since then is the AUTHORS which was updated in 2007. There are 35 open bugs against mcrypt in SourceForge the oldest dating from 2003"
http://thefsb.tumblr.com/post/110639027905/custodians-of-php-vote-to-keep-a-crypto-lib
nice post
ReplyDeletePenyebab Penyakit Miom