Thursday, April 30, 2015

notes of yara 3.4.0 compiling

tools & version:

  • windows 8.1
  • visual studio 2013
  • yara library 3.3.0 or 3.2.0


foreword

Just notes of yara compiling process on windows with visual studio. I will compile without CUCKOO support - bcs I don't need this & don't want to compile yet another library (jansson64.lib)


action


1) unpack archive

2) open solution: .\yara-3.3.0\windows\yara\yara.sln

3) open 'utils.h' -> replace '#define YR_API EXTERNC __declspec(dllexport)' to '#define YR_API EXTERNC' (bcs I don't like exported symbols in my exe files, and link I wanna statically)

4) choose platform & mode

5) set runtime library for all projects (yara & yarac & libyara):
    properties -> c/c++ -> code generation -> runtime library -> /MTd for debug or /MT for release
    (you can select several projects in time - using 'ctrl'+left_mouse_button_click)

6) add to "Preprocessor Definitions" of all projects
    (Properties -> Configuration Properties -> C/C++ -> Preprocessor -> Preprocessor Definitions)
    lines to not conflict with mysql c connector, for example:
strlcpy=libyara_internal_strlcpy
strlcat=libyara_internal_strlcat


7) Go to libyara properties:
    Properties -> Configuration Properties -> C/C++ -> Preprocessor -> Preprocessor Definitions
    and delete CUCKOO from this list.
Then go to libyara properties:
    Properties -> Configuration Properties -> Librarian -> General -> Additional Dependencies
    and delete jansson64.lib from this list.

7) Here are you must choose - you want to compile it with openssl or without.
Why do you need openssl in yara library:
    - Generate an import hash: https://www.mandiant.com/blog/tracking-malware-import-hashing/ (uses define HAVE_LIBCRYPTO)
    - PE module of yara can extract some info from pe digital signature certificate. (uses define HAVE_LIBCRYPTO)
#if defined(HAVE_LIBCRYPTO)
begin_struct_array("signatures");
  declare_string("issuer");
  declare_string("subject");
  declare_integer("version");
  declare_string("algorithm");
  declare_string("serial");
  declare_integer("not_before");
  declare_integer("not_after");
  declare_function("valid_on", "i", "i", valid_on);
  end_struct_array("signatures");
declare_integer("number_of_signatures");
#endif   

    - HASH module of yara can calc provide you cryptographic hash functions: md5, sha1, sha256, checksum32 (uses define HASH, appeared in 3.3.0 version)
   
If you need some of this functionality - you need to build openssl & you need add for all projects:
    - Additional library directory
    - library file of openssl (libeay32.lib on my pc)

If you don't need this functionality
    - delete HAVE_LIBCRYPTO & HASH from "Preprocessor Definitions" of libyara, and insert HAVE_TIMEGM line - else you get undefined type 'tm'.
        Properties -> Configuration Properties -> C/C++ -> Preprocessor -> Preprocessor Definitions
    - delete libeay64.lib from
        Properties -> Configuration Properties -> Librarian -> General -> Additional Dependencies

After that everything will compiles fine.


Possible errors for googling people


warnings:

---------------------------------------------------------------------------------------------
 in x86 (32-bits mode) you will get next warnings:
    3>args.obj : warning LNK4075: ignoring '/EDITANDCONTINUE' due to '/OPT:LBR' specification
    2>args.obj : warning LNK4075: ignoring '/EDITANDCONTINUE' due to '/OPT:LBR' specification

---------------------------------------------------------------------------------------------
1>D:\blablabla\yara-3.3.0\libyara\include\yara/object.h(23): warning C4005: 'INFINITY' : macro redefinition
1>          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\math.h(102) : see previous definition of 'INFINITY'
1>D:\blablabla\yara-3.3.0\libyara\include\yara/object.h(24): warning C4005: 'NAN' : macro redefinition
1>          C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\include\math.h(106) : see previous definition of 'NAN'

---------------------------------------------------------------------------------------------
1>libeay32.lib(fips_ers.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library
1>libeay32.lib(ecp_nistp224.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library
1>libeay32.lib(ecp_nistp256.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library
1>libeay32.lib(ecp_nistp521.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library
1>libeay32.lib(ecp_nistputil.obj) : warning LNK4221: This object file does not define any previously undefined public symbols, so it will not be used by any link operation that consumes this library

---------------------------------------------------------------------------------------------
what to do - ignore them


if you didn't set HAVE_TIMEGM

---------------------------------------------------------------------------------------------
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(23): error C2027: use of undefined type 'tm'
1>          d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(14) : see declaration of 'tm'
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(23): error C2227: left of '->tm_year' must point to class/struct/union/generic type
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(26): error C2027: use of undefined type 'tm'
1>          d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(14) : see declaration of 'tm'
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(26): error C2227: left of '->tm_mon' must point to class/struct/union/generic type
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(27): error C2027: use of undefined type 'tm'
1>          d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(14) : see declaration of 'tm'
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(27): error C2227: left of '->tm_year' must point to class/struct/union/generic type
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(29): error C2027: use of undefined type 'tm'
1>          d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(14) : see declaration of 'tm'
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(29): error C2227: left of '->tm_mday' must point to class/struct/union/generic type
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(31): error C2027: use of undefined type 'tm'
1>          d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(14) : see declaration of 'tm'
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(31): error C2227: left of '->tm_hour' must point to class/struct/union/generic type
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(33): error C2027: use of undefined type 'tm'
1>          d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(14) : see declaration of 'tm'
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(33): error C2227: left of '->tm_min' must point to class/struct/union/generic type
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(35): error C2027: use of undefined type 'tm'
1>          d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(14) : see declaration of 'tm'
1>d:\blablabla\yara-3.2.0\libyara\modules\pe_utils.c(35): error C2227: left of '->tm_sec' must point to class/struct/union/generic type

---------------------------------------------------------------------------------------------
what to do - add HAVE_TIMEGM

Wednesday, April 29, 2015

linking problem with C

Story


Recently I've got errors during building project (at linking step), spent a lot of time to solve it - wrote some perl script for solving my problem, but finally found simple and nice solution.
The problem occured when several .lib files stored the same symbol - Visual Studio in this case generates next message:
    1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: socket already defined in ws2_32.lib(WS2_32.dll)
that means:
    - library file Win32_Interop.lib contains symbol 'socket'
    - library file ws2_32.lib contains symbol 'socket'
and linker can't decide which instance must be connected when somewhere called socket().

Examples of such errors (which I met):
    - yara defines symbols (strlcpy,strlcat) which is also defined in mysqlclient.lib
    - redis defines symbols (pthread_create, pthread_cond_init, pthread_cond_destroy, pthread_cond_wait, pthread_cond_signal) which is also defined in mysqlclient.lib
    - redis defines symbols (socket, WSASend, WSARecv, WSACleanup, ioctlsocket, setsockopt, getsockopt, connect, listen, bind, shutdown, htons, htonl, getpeername, getsockname, ntohs, select, ntohl, freeaddrinfo, getaddrinfo, WSASetLastError, WSAGetLastError, WSAIoctl) which is also defined in standard windows library ws2_32.lib

There are simple way - say to linker smthng like "choose them in random way":
'properties -> Linker -> General -> Force File Output' with any of next values:
    "Enabled (/FORCE)"
    "Multiply Defined Symbol Only (/FORCE:MULTIPLE)"
but it can lead to some mystic bugs. So, I don't like that way.

Here are facts which obviously lead to solution:
    - All such disambiguated symbols which I saw - it's only for internal usage in libraries. No one such symbol designed for calling outside.
    - And all such symbols I saw - was defined in one file, and called from another - so, I couldn't just make them 'static'.
    - All these projects I'm building from source code.
And here are obvious solution - if it's internal library stuff - who cares which names these internal symbols have.

So, task became 'replace all instances of lexem in projects to another lexem'.
It's refactoring task - and seems I can use typical refactoring tools. but here are things:
    - paid software (klocwork, Slickedit) - I need to use it only once to create lib file. I don't want to pay for that. And trial sucks.
    - some plugins for some IDE (Eclipse CDT, Xrefactory for Emacs, Netbeans, Qt Creator) - I don't think they have batch mode. I don't want manually click for refactoring. And I don't want install some huge ide.
    - GCC MELT - maybe, but I don't want gcc.
Actually one tool looked nice - cscope - but on the first meeting it looked creepy. And I still didn't find out how it works.

I searched ways to solve it by myself.
Simple s/lexem_A/lexem_B/ obviously won't work because if you want replace WSACleanup in redis to any another lexem - it ruins next constructions:
    auto f_WSACleanup = dllfunctor_stdcall<int>("ws2_32.dll", "WSACleanup");
because string constant will be replaced also.
Also if lexem_A is substring of some another lexem - it also will be affected.

Make lexical analysis is difficult. Especially if you are newbie in this area.

On that time I wrote perl script, which uses some simple heuristic analysis of context. Tested it on 2 projects - it worked fine. But here are simple & nice solution.


Simple and nice solution

Answet is preprocessing.
C/C++ compiler apply defines to lexems - it won't affect on substrings of another lexems; it won't affect on string constants. Ideal solution.
Even more - Visual Studio allows set defines without modifications of any source code. And even more - Visual Studio allows you do it with several projects in time! You can set defines for all projects in solution.
Just select c/c++ projects by clicking on projects with pressed 'ctrl' button (if you select also c# project, for example - nothing will work. Only c/c++ projects must be selected).
Then click right mouse button and go to
    "Properties" -> "Configuration Properties" -> "C/C++" -> "Preprocessor" -> "Preprocessor Definitions"
and add what do you want to replace in format lexemA=lexemB. For example - here are my stuff for yara (to not conflict with mysqlclient.lib):
strlcpy=libyara_internal_strlcpy
strlcat=libyara_internal_strlcat


That's it. Apply it & compile - all lexems will be replaced.


Examples

--------------------------------------------------------
redis (don't forget you don't need select c# project - "ReleasePackagingTool"):
pipe=redis_internal_pipe
socket=redis_internal_socket
fdapi_close=redis_internal_fdapi_close
open=redis_internal_open
setsockopt=redis_internal_setsockopt
fcntl=redis_internal_fcntl
poll=redis_internal_poll
getsockopt=redis_internal_getsockopt
connect=redis_internal_connect
read=redis_internal_read
write=redis_internal_write
fsync=redis_internal_fsync
fdapi_fstat64=redis_internal_fdapi_fstat64
listen=redis_internal_listen
ftruncate=redis_internal_ftruncate
bind=redis_internal_bind
shutdown=redis_internal_shutdown
htons=redis_internal_htons
htonl=redis_internal_htonl
getpeername=redis_internal_getpeername
getsockname=redis_internal_getsockname
ntohs=redis_internal_ntohs
ioctlsocket=redis_internal_ioctlsocket
inet_addr=redis_internal_inet_addr
gethostbyname=redis_internal_gethostbyname
inet_ntoa=redis_internal_inet_ntoa
fdapi_fwrite=redis_internal_fdapi_fwrite
fdapi_setmode=redis_internal_fdapi_setmode
WSASetLastError=redis_internal_WSASetLastError
WSAGetLastError=redis_internal_WSAGetLastError
WSAIoctl=redis_internal_WSAIoctl
WSASend=redis_internal_WSASend
WSARecv=redis_internal_WSARecv
WSACleanup=redis_internal_WSACleanup
WSAGetOverlappedResult=redis_internal_WSAGetOverlappedResult
WSADuplicateSocket=redis_internal_WSADuplicateSocket
WSASocket=redis_internal_WSASocket
select=redis_internal_select
ntohl=redis_internal_ntohl
isatty=redis_internal_isatty
access=redis_internal_access
lseek64=redis_internal_lseek64
fdapi_get_osfhandle=redis_internal_fdapi_get_osfhandle
fdapi_open_osfhandle=redis_internal_fdapi_open_osfhandle
freeaddrinfo=redis_internal_freeaddrinfo
getaddrinfo=redis_internal_getaddrinfo
inet_ntop=redis_internal_inet_ntop
accept=redis_internal_accept
pthread_self=redis_internal_pthread_self
pthread_create=redis_internal_pthread_create
pthread_cond_init=redis_internal_pthread_cond_init
pthread_cond_destroy=redis_internal_pthread_cond_destroy
pthread_cond_wait=redis_internal_pthread_cond_wait
pthread_cond_signal=redis_internal_pthread_cond_signal

--------------------------------------------------------
yara:
strlcpy=libyara_internal_strlcpy
strlcat=libyara_internal_strlcat

--------------------------------------------------------


Script

Just put it here: https://github.com/HostageBrain/my_lovely_refactor
It analyzes context and replaces lexems which you needed.
Usage:
    - you are specifying extensions
    - you are specifying lexems for replacing
    - you can optional add custom data types for heuristic (redis required 2 such items)
    - you are running 'my_lovely_refactor.pl path_to_directory'
And it prints files where replacings happened. Original file saved in .N.bak extension, where N - nearest free number.


Errors for googling people

if you are trying compile together redis, mysql & ws2_32
--------------------------------------------------------
1>------ Build started: Project: blablabla, Configuration: Debug x64 ------
1>Win32_Interop.lib(win32fixes.obj) : error LNK2005: pthread_create already defined in mysqlclient.lib(my_winthread.obj)
1>Win32_Interop.lib(win32fixes.obj) : error LNK2005: pthread_cond_init already defined in mysqlclient.lib(my_wincond.obj)
1>Win32_Interop.lib(win32fixes.obj) : error LNK2005: pthread_cond_destroy already defined in mysqlclient.lib(my_wincond.obj)
1>Win32_Interop.lib(win32fixes.obj) : error LNK2005: pthread_cond_wait already defined in mysqlclient.lib(my_wincond.obj)
1>Win32_Interop.lib(win32fixes.obj) : error LNK2005: pthread_cond_signal already defined in mysqlclient.lib(my_wincond.obj)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: socket already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSASend already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSARecv already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSACleanup already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: ioctlsocket already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: setsockopt already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: getsockopt already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: connect already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: listen already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: bind already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: shutdown already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: htons already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: htonl already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: getpeername already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: getsockname already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: ntohs already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: select already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: ntohl already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: freeaddrinfo already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: getaddrinfo already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSASetLastError already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSAGetLastError already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSAIoctl already defined in ws2_32.lib(WS2_32.dll)
1>D:\projects\blablabla\blablabla.exe : fatal error LNK1169: one or more multiply defined symbols found
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========


--------------------------------------------------------
what to do - use defines (read note from the beginning)


if you are trying compile together mysql C connector & yara
--------------------------------------------------------
1>------ Build started: Project: blablabla, Configuration: Debug x64 ------
1>libyara64.lib(strutils.obj) : error LNK2005: "unsigned __int64 __cdecl strlcat(char *,char const *,unsigned __int64)" (?strlcat@@YA_KPEADPEBD_K@Z) already defined in mysqlclient.lib(crypt_genhash_impl.obj)
1>D:\projects\blablabla\blablabla.exe : fatal error LNK1169: one or more multiply defined symbols found
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========


--------------------------------------------------------
what to do - use defines (read note from the beginning)

Wednesday, April 22, 2015

build redis3m on windows

Foreword

Recently I've needed to build redis c++ client on windows with support of EVAL command.
There are exists some client:
    https://github.com/mrpi/redis-cplusplus-client
and here are it's port to Windows:
    http://sourceforge.net/p/rediscplusplusc
but it had last update "2012-07-24", and eval support in server appeared only in september of 2012. So, it's not my case. redis3m looks good for my case.


tools & versions

  • Windows 8.1
  • Visual Studio 2013 professional
  • Boost 1.57 in directory D:\projects\libraries\boost_1_57_0 with libraries in D:\projects\libraries\boost_1_57_0\stage-x64
Will suppose we are working in D:\tmp dir


action

1) Take "Microsoft Open Technologies" redis (it takes ~3 minutes) (exec in D:\tmp)
    git clone https://github.com/MSOpenTech/redis.git
    Now the last version is "Redis 2.8.19"

2) Open solution
    D:\tmp\redis\msvs\RedisServer.sln
    Choose platform which you need & Build.
    Also here probably you want to insert some defines to get rid of linking errors - [link]
3) Take redis3m (exec in D:\tmp)
    git clone https://github.com/luca3m/redis3m.git
4) Open solution
    D:\tmp\redis3m\Redis3M-VisualStudio\Redis3M-VisualStudio.sln
5) Go to "Redis3M-VisualStudio" -> "properties" -> C/C++ -> General -> "Additional Include Directories"
    replace "E:\github\redis\deps" to "D:\tmp\redis\deps"
6) You need to do something with ERROR bcs in wingdi.h there are:
    /* Region Flags */
    #define ERROR 0
and then you have in reply.h:
enum class type_t
{
    STRING = 1,
    ARRAY = 2,
    INTEGER = 3,
    NIL = 4,
    STATUS = 5,
    ERROR = 6
};


so, one of variants - you can just #undef ERROR. go to files
  • reply.cpp
  • connection.cpp
and after "#include <hiredis/hiredis.h>" insert:
#ifdef ERROR
#undef ERROR
#endif


(another solution - replace every occurrence of type_t::ERROR on something anohter - for example on type_t::MY_LOVELY_ERROR)

7) Build "Redis3M-VisualStudio"

And now you have everything what you need to compile stuff, which uses redis3m.


build simple.cpp example of redis3m

A) Create new project in visual studio

B) Set platform & mode which you want

C) Insert to "Properties" -> "C/C++" -> General -> "Additional Include Directories"
    "D:\tmp\redis3m\include"
    "D:\projects\libraries\boost_1_57_0"

D) Choose in "Properties" -> "C/C++" -> "Code Generation" -> "Runtime library" mode "/MTd" (for debug, or /MT for release)

E) Insert to "Properties" -> Linker -> General -> "Additional Library Directories":
for x64 debug:
    D:\tmp\redis\msvs\x64\Debug
    D:\tmp\redis3m\Redis3M-VisualStudio\Debug\x64
    D:\projects\libraries\boost_1_57_0\stage-x64
for x64 release:
    D:\tmp\redis\msvs\x64\Release
    D:\tmp\redis3m\Redis3M-VisualStudio\Release\x64
    D:\projects\libraries\boost_1_57_0\stage-x64

F) Insert to "Properties" -> Linker -> Input -> "Additional Dependencies":
    redis3m.lib
    Win32_Interop.lib
    hiredis.lib

G) Take "simple.cpp" example from redis3m examples
---------------------------------------------------------
#include <redis3m/redis3m.hpp>
#include <iostream>

using namespace redis3m;

int main(int argc, char **argv)
{
        connection::ptr_t conn = connection::create();
        conn->run(command("SET") << "foo" << "bar" );
        reply r = conn->run(command("GET") << "foo" );   
        std::cout << "FOO is: " << r.str() << std::endl;
}
---------------------------------------------------------

it must compiles


possible errors (for googling people):

If you didn't specify path to 'hiredis/hiredis.h':
-----------------------------------------------------------------------------------------------------------------------------------
1>d:\tmp\redis3m\src\reply.cpp(4): fatal error C1083: Cannot open include file: 'hiredis/hiredis.h': No such file or directory
1>d:\tmp\redis3m\src\connection.cpp(5): fatal error C1083: Cannot open include file: 'hiredis/hiredis.h': No such file or directory
-----------------------------------------------------------------------------------------------------------------------------------
what to do: go to step 5

If you didn't solve problem with ERROR in wingdi.h:
-----------------------------------------------------------------------------------------------------------------------------------
1>d:\tmp\redis3m\include\redis3m\reply.h(30): error C2059: syntax error : 'constant'
1>d:\tmp\redis3m\include\redis3m\reply.h(30): error C3805: 'constant': unexpected token, expected either '}' or a ','
1>d:\tmp\redis3m\include\redis3m\reply.h(60): error C2589: 'constant' : illegal token on right side of '::'
1>d:\tmp\redis3m\include\redis3m\reply.h(60): error C2059: syntax error : '::'
1>d:\tmp\redis3m\include\redis3m\reply.h(61): error C2143: syntax error : missing ';' before '{'
1>d:\tmp\redis3m\include\redis3m\reply.h(64): error C2181: illegal else without matching if
1>d:\tmp\redis3m\src\reply.cpp(10): error C2589: 'constant' : illegal token on right side of '::'
1>d:\tmp\redis3m\src\reply.cpp(10): error C2143: syntax error : missing ')' before '::'
1>d:\tmp\redis3m\src\reply.cpp(10): error C2612: trailing '::' illegal in base/member initializer list
1>d:\tmp\redis3m\src\reply.cpp(32): fatal error C1004: unexpected end-of-file found
1>d:\tmp\redis3m\src\connection.cpp(54): error C2589: 'constant' : illegal token on right side of '::'
1>d:\tmp\redis3m\src\connection.cpp(54): error C2059: syntax error : '::'
1>d:\tmp\redis3m\src\connection.cpp(56): error C2143: syntax error : missing ';' before '{'
-----------------------------------------------------------------------------------------------------------------------------------
what to do: go to step 6 in the note


Thursday, April 9, 2015

using GYP/depot_tools and building pdfium

foreword & couple words of GYP

Recently I've needed to build pdfium - that engine, which is rendering pdf in chrome browser. So, I've found these links:
  • 'Building PDFium' instruction [link]
and on this page was link to 'depot_tools'
  • 'Install depot_tools' instruction [link]
So, what's that. Google fellows made great universal cross-platform building system - GYP (generalize your project). It's like cmake, and for cases with I encounted - I liked it much more that cmake (my subjective opinion). 'depot_tools' - it is GYP.



problem

And here was a problem - 'Non-cygwin' part of documentation is obsolete and if you make everything what documentation says - nothing will work. And even if you succsseed in documentation part - you will get a lot of not described problems.
In this note I will describe solving this problems.


tools & versions

I have installed on my pc:
  • Windows 8.1
  • Visual Studio 2013 professional
  • git version 1.9.4.msysgit.0
  • All filesystem/archiving operations I'm doing with FAR Manager 3.0


problem solving in a short

0. Create 'c:\depot_tools' directory (or any another you want) & add it to %PATH%

1. Forget about archive with tools in documentation - take last version from git:
    >cd c:\
    >git clone https://chromium.googlesource.com/chromium/tools/depot_tools
    >cd ./depot_tools

2. Make some git magic (without it nothing will work)
    >git reset --hard HEAD

3. run 'gclient'
    When it asks you
        Your depot_tools checkout is configured to fetch from an obsolete URL
        Would you like to update it? [y/N]:
    answer Y
    If everything is ok - you will get something like that, without any errors:


4. This step probably optional. Your python - 'C:\depot_tools\python276_bin' doesn't have gyp module. Last time it generated errors, but not now. I don't know why. So, if you have errors bcs of absent 'gyp' module or just wanna have it 'just in case' - install it.

    somewhere in temp directory:
    >git clone https://chromium.googlesource.com/external/gyp
    >C:\depot_tools\python276_bin\python.exe setup.py install

And here you have working 'depot_tools'. And next steps for those who wants to build pdfium (as bonus, you will get ready 'v8' engine as dependency).

5. Somewhere where you want to put pdfium sources (for example in "D:\tmp"), exec next commands (it will download ~600 Mbytes of sources):
    >mkdir pdfium
    >cd pdfium
    >gclient config --name . --unmanaged https://pdfium.googlesource.com/pdfium.git
    >gclient sync

Now you have project pdfium.

6. To build .sln files exec:
    >C:\depot_tools\python276_bin\python.exe D:\tmp\pdfium\build\gyp_pdfium

    You will see something like that:
--------------------------------------
Updating projects from gyp files...
Warning: Missing input files:
D:\tmp\pdfium\testing\gtest-support.h
D:\tmp\pdfium\testing\gmock_mutant.

-------------------------------------- 
    if it so - everything is fine. Everything will build even without these files.

7. During building process, v8 engine expects to find python in 'D:\tmp\pdfium\v8\third_party\python_26'. So, create this directory and copy to it content of your 'C:\depot_tools\python276_bin'

8. If you will try to build pdfium for x86 platform and see something like that:
        LINK : warning LNK4068: /MACHINE not specified; defaulting to X64
    find 'pdfium' in solutions list, go to
    pdfium -> configuration properties -> librarian -> all options
    find 'Target machine' option with empty value
    set (from dropdown):
        MachineX86 (/MACHINE:X86)
    after that on my machive I have:
        ========== Build: 37 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

9. For building for x86-64 target platform you need to generate new .sln files:
    >C:\depot_tools\python276_bin\python.exe D:\tmp\pdfium\build\gyp_pdfium -Dtarget_arch=x64
    then it builds easily


illustration of problems with following obsoleted instructions

[this part mostly for people, googling solution by error strings]
[it's illustration how you don't need to do - how you need to do you can find in part 'problem solving in a short']

Let's try follow instruction - I will follow 'non-Cygwin' branch of instruction:

1. Get a copy of depot_tools:

This 'install-depot-tools' link has really detailed instruction and zip archive with 'depot_tools' for downloading. So, download archive 'depot_tools.zip' and extract it, for exaple, to 'C:\depot_tools' (for exaple, with FAR manager)

2. Add depot_tools to the end (not start!) of your PATH ('With Administrator access' branch)

3. Non-cygwin: Run gclient from the cmd shell
so, enter in FAR 'gclient' (it will start c:\depot_tools\gclient.bat) and wait.

>gclient
Installing python 2.7.6...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/python276_bin.zip
Installing git 1.9.0.chromium.5 (avg 1-2 min download) ...
Fetching from https://commondatastorage.googleapis.com/chrome-infra/git-1.9.0.chromium.5_bin.zip
""c:\depot_tools\git.bat"" is not recognized as an internal or external command
""c:\depot_tools\git.bat"" is not recognized as an internal or external command


so, we have the same problem which that guy - link.
and you dive into this problem and find solution like described here - link.

'C:\depot_tools\bootstrap\win\win_tools.bat' from 'depot_tools.zip' contain next lines:
------------------------------------------------------------------------------------------------------
:: Ensure autocrlf and filemode are set correctly.
call "%WIN_TOOLS_ROOT_DIR%\git.bat" config --system core.autocrlf false
call "%WIN_TOOLS_ROOT_DIR%\git.bat" config --system core.filemode false
goto :GIT_COPY_BATCH_FILES


:GIT_COPY_BATCH_FILES
:: Create the batch files.
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\git.bat" "%WIN_TOOLS_ROOT_DIR%\git.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\gitk.bat" "%WIN_TOOLS_ROOT_DIR%\gitk.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\ssh.bat" "%WIN_TOOLS_ROOT_DIR%\ssh.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\ssh-keygen.bat" "%WIN_TOOLS_ROOT_DIR%\ssh-keygen.bat" 1>nul
goto :SVN_CHECK
------------------------------------------------------------------------------------------------------


Firsly it's trying to call git.bat, and only after that create this file. After exec 'gclient' this file will be replaced by new version where this fixed, but when you see this error - executed old, bugged version of this file.
So, you decided to make something like that - unpack 'depot_tools' in 'C:\depot_tools' one more time and replace wrong lines by correct ones:

------------------------------------------------------------------------------------------------------
del "%ZIP_DIR%\git.zip"
goto :GIT_COPY_BATCH_FILES


:GIT_COPY_BATCH_FILES
:: Create the batch files.
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\git.bat" "%WIN_TOOLS_ROOT_DIR%\git.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\gitk.bat" "%WIN_TOOLS_ROOT_DIR%\gitk.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\ssh.bat" "%WIN_TOOLS_ROOT_DIR%\ssh.bat" 1>nul
call copy /y "%WIN_TOOLS_ROOT_DIR%\%GIT_BIN_DIR%\ssh-keygen.bat" "%WIN_TOOLS_ROOT_DIR%\ssh-keygen.bat" 1>nul
:: Ensure autocrlf and filemode are set correctly.
call "%WIN_TOOLS_ROOT_DIR%\git.bat" config --system core.autocrlf false
call "%WIN_TOOLS_ROOT_DIR%\git.bat" config --system core.filemode false
goto :SVN_CHECK
------------------------------------------------------------------------------------------------------


so, after that you will get next log:

------------------------------------------------------------------------------------------------------
C:\depot_tools>gclient
Installing python 2.7.6...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/python276_bin.zip
Installing git 1.9.0.chromium.5 (avg 1-2 min download) ...
Fetching from https://commondatastorage.googleapis.com/chrome-infra/git-1.9.0.chromium.5_bin.zip
Installing subversion ...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/svn_bin.zip
Cannot rebase: You have unstaged changes.
Please commit or stash them.
Failed to update depot_tools.
Usage: gclient.py <command> [options]

Meta checkout manager supporting both Subversion and GIT.

Commands are:
  cleanup  cleans up all working copies
  config   creates a .gclient file in the current directory
  diff     displays local diff for every dependencies
  fetch    fetches upstream commits for all modules
  grep     greps through git repos managed by gclient
  help     prints list of commands or help for a specific command
  hookinfo outputs the hooks that would be run by `gclient runhooks`
  pack     generates a patch which can be applied at the root of the tree
  recurse  operates [command args ...] on all the dependencies
  revert   reverts all modifications in every dependencies
  revinfo  outputs revision info mapping for the client and its dependencies
  runhooks runs hooks for files that have been modified in the local working copy
  status   shows modification status for every dependencies
  sync     checkout/update all modules

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -j JOBS, --jobs=JOBS  Specify how many SCM commands can run in parallel;
                        defaults to 8 on this machine
  -v, --verbose         Produces additional output for diagnostics. Can be
                        used up to three times for more logging info.
  --gclientfile=CONFIG_FILENAME
                        Specify an alternate .gclient file
  --spec=SPEC           create a gclient file containing the provided string.
                        Due to Cygwin/Python brokenness, it can't contain any
                        newlines.
  --no-nag-max          Ignored for backwards compatibility.
------------------------------------------------------------------------------------------------------


what the fuck?
    Cannot rebase: You have unstaged changes.
    Please commit or stash them.
    Failed to update depot_tools.

so, you start search solution, find links:
and you find in 'update_depot_tools.bat' line:
    call git config remote.origin.fetch > NUL

and you repeat everything one more time - unpack archive, fix lines in 'win_tools.bat', then add line to 'update_depot_tools.bat' to make next way:
    call git reset --hard HEAD > NUL
    call git config remote.origin.fetch > NUL

and finally you get this problem - https://code.google.com/p/chromium/issues/detail?id=448981

------------------------------------------------------------------------------------------------------
Installing python 2.7.6...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/python276_bin.zip
Installing git 1.9.0.chromium.5 (avg 1-2 min download) ...
Fetching from https://commondatastorage.googleapis.com/chrome-infra/git-1.9.0.chromium.5_bin.zip
'"Z:\Projects\Chromium\depot_tools\git.bat"' is not recognized as an internal or external command,
operable program or batch file.
'"Z:\Projects\Chromium\depot_tools\git.bat"' is not recognized as an internal or external command,
operable program or batch file.
Installing subversion ...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/svn_bin.zip
'*' is not recognized as an internal or external command,
operable program or batch file.

------------------------------------------------------------------------------------------------------

and on that moment I tired from this shit and searched another way to install it. And here are some more error messages, which appearing when you are dealing with GYP, depot_tools & pdfium.

------------------------------------------------------------------------------------------------------
Installing python 2.7.6...
Fetching from https://src.chromium.org/svn/trunk/tools/third_party/python276_bin.zip
Installing git 1.9.5.chromium.6 (avg 1-2 min download) ...
Fetching from https://commondatastorage.googleapis.com/chrome-infra/git-1.9.5.chromium.6_bin.zip
[-] XMLHTTP 80072ee2: Cannot make HTTP request (The operation timed out

... Failed to checkout git automatically.
You should get the "prebaked" version used at https://commondatastorage.googleapis.com/chrome-infra/git-1.9.5.chromium.6_bin.zip
------------------------------------------------------------------------------------------------------

what to do: you forgot to make 'git reset --hard HEAD' - so, step 2 in tutorial above.

during building pdfium you can find next error:
------------------------------------------------------------------------------------------------------
1>  bash: python: command not found
1>C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V120\Microsoft.CppCommon.targets(170,5): error MSB6006: "cmd.exe" exited with code 127.
------------------------------------------------------------------------------------------------------

what to do - it can't find python in pdfium - so, step 7 in tutorial above.

if you are trying to build x64 target platform from x32 sources:
------------------------------------------------------------------------------------------------------
1>cl : Command line warning D9002: ignoring unknown option '/arch:SSE2'
1>D:\tmp\pdfium\v8\src/base/build_config.h(115): fatal error C1189: #error :  Target architecture ia32 is only supported on ia32 host (..\..\src\assert-scope.cc)
------------------------------------------------------------------------------------------------------

what to do - you need to generate another visual studio files - so, step 9 in tutorial above.

Tuesday, April 7, 2015

blowfish in cfb mode in openssl & mcrypt

foreword

Some time ago I've needed to make client for one client-server protocol, where one step included blowfish-cfb encryption/decryption of data. Server side was written in PHP and used mcrypt library for encryption (need to note - mcrypt died in 2003). On server side it looked like that:
    $result = mcrypt_decrypt(MCRYPT_BLOWFISH, $key, $data, MCRYPT_MODE_CFB);
    $result = mcrypt_encrypt(MCRYPT_BLOWFISH, $key, $data, MCRYPT_MODE_CFB);
and that was bad code, bcs there are no 5th parameter. It set indirectly to \0\0\0\0\0\0\0\0. and it worked only before PHP 5.6 - after it won't work. In documentation written: "Invalid key and iv sizes are no longer accepted. mcrypt_encrypt() will now throw a warning and return FALSE if the inputs are invalid. Previously keys and IVs were padded with '\0' bytes to the next valid size." but this is not a problem - it's just little sad moment.

I've used c++ and OpenSSL - so, I've found description such functional in documentation (https://www.openssl.org/docs/crypto/blowfish.html):
    void BF_cfb64_encrypt(const unsigned char *in, unsigned char *out,
        long length, BF_KEY *schedule, unsigned char *ivec, int *num,
        int enc);
    "BF_cfb64_encrypt() is the CFB mode for Blowfish with 64 bit feedback."
    BF_cfb64_encrypt was the only function in openssl, which matched what I searched - blowfish & cfb

I've made encryption/decryption function using OpenSSL and found a surprise - the same data, encrypted with the same key & same 'iv' after c++&openssl and after php&mcrypt was completely different. Well, not completely - only the first byte was the same. So, it was very strange: in both cases I had some classical algothitm - blowfish - which exists more than 20 years; and some standard mode - cfb. I've spent a lot of time (something like 3 days) for understanding - I've loaded php script into php interpretator, and opened this interpretator in ollydbg, and debugged debugged debugged .

Openssl and mcrypt - both of these libraries - it's like framework, where you can connect primitives, but with some difference:
  • in mcrypt you have privitive of CFB and block cipher Blowfish - and they somehow connected during compiling & runtime - so, in sources you can't understand how it connected.
  • in openssl you have BF_cfb64_encrypt, but ciphers can be connected into chains - and it happens also during compiling & runtime.
so, for this case I debugged in ollydbg only mcrypt (as part of php interpreter), but such modular architecture in openssl forced me debug it into previous note.

I wanted to write only this note, but during this research appeared 3 more notes.
And I've read a lot of discussions of this problem - and didn't find one simple answer. So, that's why I decided to write of this problem.


problem

When I encrypt data with blowfish in cfb mode on openssl - I can't decrypt on mcrypt correctly.
and vice verca:
When I encrypt data with mcrypt in cfb mode on openssl - I can't decrypt on openssl correctly.

It looked like these libraries incompatible, despite they implement the same.

This problem affects a lot of languages (I've seen bindings to: c/c++, perl, python, php, ruby, rust, nodejs, java, even haskell, lisp, lua, go, rust and some languages which I've not heard - nim, racket), because openssl & mcrypt - ones of the most famous crypto libraries, and any wide-spreaded programming language have bindings/wrappers/modules to these libraries. And if someone writes it's own implementation of blowfish-cfb - this person will likely check is it work correct, in comparison with one of these libraries. Such examples:
  • C# bouncycastle library (behave as openssl)
  • python crypto (behave as mcrypt by default)

Last commit to mcrypt project was in 2003 [link]. You shouldn't use it in your new projects.

to pass lyrics and just get solution - just scroll to "problem solve in a short".


illustration of a problem

php mcrypt
#--------------------------------------------------------------
<?php
    $e = mcrypt_encrypt(MCRYPT_BLOWFISH, "mypass", "my_lovely_text", MCRYPT_MODE_CFB, "\0\0\0\0\0\0\0\0");
    echo "$e";
?>
#--------------------------------------------------------------

    3C 38 BA 2E-46 C4 2A 55-38 07 1F DE-89 E3

perl openssl
#--------------------------------------------------------------
use Crypt::OpenSSL::Blowfish::CFB64;
my $crypt = Crypt::OpenSSL::Blowfish::CFB64->new('mypass', pack( C8 => 0,0,0,0,0,0,0,0 ));
my $binary_data = $crypt->encrypt("my_lovely_text");
print $binary_data;
#--------------------------------------------------------------

    3C 49 42 3A-0F 4F 65 50-67 29 74 36-31 37

C/C++ openssl:
//------------------------------------------------------------
#include <string.h>
#include <openssl/blowfish.h>

int main(int argc, char** argv)
{
    const unsigned char plaintext[] = "my_lovely_text";
    const unsigned char key[] = "mypass";

    unsigned char out_buf[1024];

    int counter = 0;
    unsigned char iv[8];
    BF_KEY keyStruct;

    memset(iv, 0, 8);
    BF_set_key(&keyStruct, strlen((const char*)key), key);
    BF_cfb64_encrypt(
        plaintext,
        out_buf,
        strlen((const char*)plaintext),
        &keyStruct,
        iv,
        &counter,
        BF_ENCRYPT
        );
   
    std::ofstream o("file.bf", std::ios::binary);
    o.write((char*)out_buf, plaintext_len);
    o.close();
   
    return 0;
}
//------------------------------------------------------------

    3c 49 42 3a 0f 4f 65 50 67 29 74 36 31 37


same_problems

I've found in the Internet several similar problems:
    0) https://github.com/tugrul/node-mcrypt
        The author writes:
        "There is already OpenSSL extension bundled with Node.js but there are something wrong with some traditional encryption algorithms on OpenSSL"
        Someone tired from this discrepancy and made his own wrapper on mcrypt.
        This is not looks like answer for question 'why?'.
    1) here are some Russian guy with such problem
        http://phpclub.ru/talk/threads/delphi-php-mcrypt-blowfish.53690/
        http://www.sql.ru/forum/544138/delphi-php-mcrypt-blowfish
        Nodoby proposed good hypothesis.
    2) here are also some Russian guy with such problem
        http://ru-root.livejournal.com/2759003.html
        and solution he found - use mcrypt instead of openssl :D
        But he noticed that the 1st byte is the same. When I dealt with that - I didn't notice that.
        And actually he was the most closest to the solving problem.
        And in comments there was right answer, but I missed it when googled answer.
    3) here are encrypt data with php and trying decrypt with nodejs, also check if everything is correct with openssl console tool (this theme I've mentioned here)
        http://stackoverflow.com/questions/8895370/decrypting-mcrypt-encoded-text-with-node-js
       
so, everybody adviced:
    - use some padding
    - use iv
    - use for decryption the same library which encrypted data
and nobody seen the reason - why it happens? :-/


lyrics

to make long story short - I will pass my adventures with olly debugger & just leave here source of
  • openssl blowfish encryption in cfb mode, and 
  • generic cfb algorithm - from _mcrypt, where _mcrypt_block_encrypt will be blowfish block encryption.

// openssl blowfish cfb
// openssl-1.0.1j\crypto\bf\bf_cfb64.c

#define BF_ENCRYPT      1
#define BF_DECRYPT      0

void BF_cfb64_encrypt(const unsigned char *in, unsigned char *out, long length,
         const BF_KEY *schedule, unsigned char *ivec, int *num, int encrypt)
    {
    register BF_LONG v0,v1,t;
    register int n= *num;
    register long l=length;
    BF_LONG ti[2];
    unsigned char *iv,c,cc;

    iv=(unsigned char *)ivec;
    if (encrypt)
        {
        while (l--)
            {
            if (n == 0)
                {
                n2l(iv,v0); ti[0]=v0;
                n2l(iv,v1); ti[1]=v1;
                BF_encrypt((BF_LONG *)ti,schedule);
                iv=(unsigned char *)ivec;
                t=ti[0]; l2n(t,iv);
                t=ti[1]; l2n(t,iv);
                iv=(unsigned char *)ivec;
                }
            c= *(in++)^iv[n];
            *(out++)=c;
            iv[n]=c;
            n=(n+1)&0x07;
            }
        }
    else
        {
        while (l--)
            {
            if (n == 0)
                {
                n2l(iv,v0); ti[0]=v0;
                n2l(iv,v1); ti[1]=v1;
                BF_encrypt((BF_LONG *)ti,schedule);
                iv=(unsigned char *)ivec;
                t=ti[0]; l2n(t,iv);
                t=ti[1]; l2n(t,iv);
                iv=(unsigned char *)ivec;
                }
            cc= *(in++);
            c=iv[n];
            iv[n]=cc;
            *(out++)=c^cc;
            n=(n+1)&0x07;
            }
        }
    v0=v1=ti[0]=ti[1]=t=c=cc=0;
    *num=n;
    }










// mcrypt blowfish cfb
// mcrypt\modules\modes\cfb.c

int _mcrypt( CFB_BUFFER* buf, void *plaintext, int len, int blocksize, void* akey, void (*func)(void*,void*), void (*func2)(void*,void*))
{                /* plaintext is 1 byte (8bit cfb) */
    char *plain = plaintext;
    int i, j;
    void (*_mcrypt_block_encrypt) (void *, void *);

    _mcrypt_block_encrypt = func;

    for (j = 0; j < len; j++) {

        memcpy(buf->enc_s_register, buf->s_register, blocksize);

        _mcrypt_block_encrypt(akey, buf->enc_s_register);

        plain[j] ^= buf->enc_s_register[0];

/* Shift the register */
        for (i = 0; i < (blocksize - 1); i++)
            buf->s_register[i] = buf->s_register[i + 1];

        buf->s_register[blocksize - 1] = plain[j];
    }

    return 0;

}





so, the problem is - it's two different algorithms!
mcrypt xor 8-bytes block and then move window to 1 byte.
openssl xor 8-bytes block and then move window to 8 byte.

and wikipedia says (http://en.wikipedia.org/wiki/Block_cipher_mode_of_operation#Cipher_Feedback_.28CFB.29)
    "This way of proceeding is known as CFB-8 or CFB-1 (according to the size of the shifting)"
As I see in source code - it often calls cfb64 and cfb8 (instead of cfb-8 and cfb-1), but the point is correct - cfb can be different.
And now, looking in sources, it's clear answer for question - "why the first byte always the same?".


solving

I already had openssl - built, included, proven. So, I've just made my own implementation of cfb8 on c++. It's mcrypt-compatible algorithm (I've tested it, of course), which imports BF_encrypt from openssl library.

#include <openssl/blowfish.h>
#include <vector>

#define endian_swap32(x) ((x << 24) | (x >> 24) | ((x&0xff00)<<8) | ((x&0xff0000)>>8))

void BlowfishEncryptCFB8(
    const std::vector<unsigned char> & in_buf,
    const std::vector<unsigned char> & key,
    const std::vector<unsigned char> & iv,
    std::vector<unsigned char> & out_buf
    )
{
    out_buf = in_buf;

    std::vector<unsigned char> internal_key(sizeof(BF_KEY));
    BF_KEY *key_ = (BF_KEY *)internal_key.data();

    BF_set_key(key_, key.size(), (const unsigned char*)key.data());

    std::vector<unsigned char> enc_s_register(8), s_register;
    s_register = iv;

    for (size_t i = 0; i < in_buf.size(); i++)
    {
        enc_s_register = s_register;

        __int32 * lp_enc_s_register = (__int32 *)enc_s_register.data();
        *lp_enc_s_register = endian_swap32(*lp_enc_s_register);
        *(lp_enc_s_register+1) = endian_swap32(*(lp_enc_s_register+1));

        BF_encrypt((BF_LONG *) enc_s_register.data(), key_);
           
        *lp_enc_s_register = endian_swap32(*lp_enc_s_register);
        *(lp_enc_s_register+1) = endian_swap32(*(lp_enc_s_register+1));

        out_buf[i] ^= enc_s_register[0];

        for (size_t j = 0; j < 7; j++)
            s_register[j] = s_register[j+1];
        s_register[7] = out_buf[i];
    }
}

void BlowfishEncryptCFB8(
    const std::vector<unsigned char> & in_buf,
    const std::vector<unsigned char> & key,
    const std::vector<unsigned char> & iv,
    std::vector<unsigned char> & out_buf
    )
{
    out_buf = in_buf;

    std::vector<unsigned char> internal_key(sizeof(BF_KEY));
    BF_KEY *key_ = (BF_KEY *)internal_key.data();

    BF_set_key(key_, key.size(), (const unsigned char*)key.data());

    std::vector<unsigned char> enc_s_register(8), s_register;
    s_register = iv;

    for (size_t i = 0; i < in_buf.size(); i++)
    {
        enc_s_register = s_register;

        __int32 * lp_enc_s_register = (__int32 *)enc_s_register.data();
        *lp_enc_s_register = endian_swap32(*lp_enc_s_register);
        *(lp_enc_s_register + 1) = endian_swap32(*(lp_enc_s_register + 1));

        BF_encrypt((BF_LONG *)enc_s_register.data(), key_);

        *lp_enc_s_register = endian_swap32(*lp_enc_s_register);
        *(lp_enc_s_register + 1) = endian_swap32(*(lp_enc_s_register + 1));

        unsigned char saved_byte = out_buf[i];
        out_buf[i] ^= enc_s_register[0];

        for (size_t j = 0; j < 7; j++)
            s_register[j] = s_register[j + 1];
        s_register[7] = saved_byte;
    }
}



problem solving in a short

  • Realize that CFB mode - it's not some concrete action - it's common name for idea use previous ciphertext as key for next block.
  • Realize that OpenSSL blowfish cfb - it's cfb64, and uses 64-bits feedback (that's why in sources this function called BF_cfb64_encrypt)
  • Realize that mcrypt blowfish cfb - it's cfb8, and uses 8-bits feedback.
  • determie which cfb mode do you have - for example, you can encode 
    • key: mypass
    • text: my_lovely_text
    • iv: 0
    • blowfish in cfb mode:
      • 3C 38 BA 2E-46 C4 2A 55-38 07 1F DE-89 E3    cfb-8 (mcrypt-like)
      • 3C 49 42 3A-0F 4F 65 50-67 29 74 36-31 37    cfb-64 (openssl-like)
  • if you are using something based on mcrypt and want openssl-compatibility:
    • search something like 'ncfb' mode - it must be openssl-compatible cfb mode.
  • else you will have simple choice:
    • find implementation of mode which you need
    • make your own implementation of cfb - block blowfish you should already have.

useful sources & links

After I've found answer by myself, I've found useful these links:

final notes

Aes has cfb8 and cfb1 modes - https://www.openssl.org/docs/apps/enc.html
    Interesting - why blowfish doesn't have such choice, I wonder?

here are comparison openssl with mcrypt:
    http://www.synet.sk/php/en/320-benchmarking-symmetric-cyphers-openssl-vs-mcrypt-in-php
    Conclusions from this link: "OpenSSL offers significantly faster algorithms. This is the right choice for heavy traffic sites."
    well, on blowfish-cfb it just makes in 8 time less work :-D

"The source code in CSV was last updated in October 2003. The only file changed since then is the AUTHORS which was updated in 2007. There are 35 open bugs against mcrypt in SourceForge the oldest dating from 2003"
    http://thefsb.tumblr.com/post/110639027905/custodians-of-php-vote-to-keep-a-crypto-lib

openssl tool gives other output that library

problem

I've encrypted some data by command-line tool openssl with symmetric cipher and can't decrypt by some another util (or even from code using openssl library).

or vice versa:

I've encrypted some data with some crypto library or some tool (not openssl) with symmetric cipher and can't decrypt by openssl command-line tool.

So, in short - why you can encrypt-decrypt (by symmetric cipher) data only by openssl tool? Why another tools failed, when trying interact with openssl command-line tool by encrypted data?

I've dealt with blowfish algorithm in cfb mode, but I think this experience can be useful also with another simmetric cipher in openssl.

to pass lyrics and just get solution - just scroll to "problem solving in a short".


light introduction in blowfish.

Blowfish - it's symmetric block cypher. Encrypt blocks 64 bits (8 bytes). [link]
To encrypt data which size diff from 8 bytes used different 'modes': ECB, CBC, CFB, OFB, CTR. Other words, 'mode' transforms block cipher to stream cipher. [link]

So, to encode with blowfish something, we need to have:
  • text
  • key
  • mode
and one more thing - initialization vector (or just 'iv'). [link]
To decrypt encrypted data you must have the same 'iv' that was used for encryption. You can consider it as second key. Or just number, which must be synchronized between Alice and Bob.


illustration of a problem (a bit of LOL):

lets try encrypt this:
  • text: my_lovely_text
  • key: mypass
  • algotithm: blowfish
  • mode: cfb
  • iv: filled by binary zeros

openssl tool 
(I know that's key usage is wrong, bcs of described further nuances. But for illustration this naive logic works fine):
    openssl.exe enc -bf-cfb -in file.txt -out file.bf -nosalt
    (it asked you password, you entered 'mypass' and it gives data):
        58 3C 1A 88-BC FF 2E 02-5C EF EA 1F-15 C9

    okay... but we forgot to set 'iv'. Who knows which 'iv' is default in openssl. Let's try one more time.
    openssl.exe enc -iv 0 -bf-cfb -in file.txt -out file.bf -nosalt
    (it asked you password, you entered 'mypass' and it gives data):
        95 3F 04 E5-FD F7 75 4E-B8 9E 39 AA-E1 2F

okay. lets see what another sources give us on the same data with the same parameters:
(I just took it from 1st page of google on request 'online encrypt blowfish cfb')

http://www.tools4noobs.com/online_tools/encrypt/
    (it doesn't have 'iv' field - so, it's some 'defaulted' value)
    (set "Encode the output using" to "HEXA"):
        302416d7806565d1a15e6b58666e

http://blowfish.online-domain-tools.com/
    here are "Init.vector" generated by password. Let's try both - default and zero iv.
    iv: e7 27 d1 46 4a e1 24 36
    result: d2    70    78    f8    0c    81    d1    9e    6b    4a    57    15    4e    33

    okay, let's try zero iv
    iv: 00 00 00 00 00 00 00 00
    result: 3c    38    ba    2e    46    c4    2a    55    38    07    1f    de    89    e3
    hmmm. And here are exists some blowfish-compat case - let's try it.

http://blowfish-compat.online-domain-tools.com/
    iv: e7 27 d1 46 4a e1 24 36
    result: 04    8a    82    b6    02    db    e3    a0    38    70    87    64    3c    e7

    okay. that was default. let's try zero iv.
    iv: 00 00 00 00 00 00 00 00
    result: 3b    3e    54    6a    51    4c    8c    a7    98    0b    0f    37    9f    c4

http://www.bierkandt.org/encryption/symmetric_encryption.php
    here are no 'iv' field - so, it must be some defaulted. Enter our data and get:
        BLOWFISH:    2cd10b1ad66f742c7db38fbab105
    okay. Buy let's try one more time (just to be sure)
        BLOWFISH:    94408e95ac0d6eac2aa3f9f61a80
    hmm... one more time?
        BLOWFISH:    4c1e679fa0fcd8e4c8e45a411462
    hmmm....
        BLOWFISH:    c2252b7a79193bd512f1e4ff2476
    looks like it returns random numbers.

http://sladex.org/blowfish.js/
    2e36059b6aa5482ffbe5fc3a7d0c3662
    (it's length 16 bytes - it's something new, because earlier every tool gives on my text, which length is 14 bytes - output 14 bytes, and it gave 16 bytes. Must be padding)


and let's try some script languages.


python crypto


from Crypto.Cipher import Blowfish
c = Blowfish.new('mypass', Blowfish.MODE_CFB, "\0"*8)
cyphertext = c.encrypt('my_lovely_text')
print (cyphertext.encode('hex'))     3c38ba2e46c42a5538071fde89e3

    WHOA - the first match! http://blowfish.online-domain-tools.com/ gave the same.


php mcrypt

<?php
$e = mcrypt_encrypt(MCRYPT_BLOWFISH, "mypass", "my_lovely_text", MCRYPT_MODE_CFB, "\0\0\0\0\0\0\0\0");
echo "$e";
?>

    3C 38 BA 2E-46 C4 2A 55-38 07 1F DE-89 E3
    Wow - yet another match.

perl openssl

use Crypt::OpenSSL::Blowfish::CFB64;
my $crypt = Crypt::OpenSSL::Blowfish::CFB64->new('mypass', pack( C8 => 0,0,0,0,0,0,0,0 ));
my $binary_data = $crypt->encrypt("my_lovely_text");
print $binary_data;


    3C 49 42 3A-0F 4F 65 50-67 29 74 36-31 37
    looks like yet another variant %)

c (well, actually a bit of c++, but who cares) openssl

#--------------------------------------------------------------
#include <stdlib.h>
#include <string.h>
#include <openssl/blowfish.h>
#include <fstream>

int main(int argc, char** argv)
{
    unsigned char out_buf[1024];

    const unsigned char plaintext[] = "my_lovely_text";
    int plaintext_len = strlen((const char*)plaintext);
    const unsigned char key[] = "mypass";

    int counter = 0;
    unsigned char iv[8];
    BF_KEY keyStruct;

    memset(iv, 0, 8);
    BF_set_key(&keyStruct, strlen((const char*)key), key);
    BF_cfb64_encrypt(
        plaintext,
        out_buf,
        plaintext_len,
        &keyStruct,
        iv,
        &counter,
        BF_ENCRYPT
        );

    std::ofstream o("file.bf", std::ios::binary);
    o.write((char*)out_buf, plaintext_len);
    o.close();
}
#--------------------------------------------------------------


    3C 49 42 3A-0F 4F 65 50-67 29 74 36-31 37
    God bless at least openssl library on different languages gives the same results)


So. Let's make summary table: (I've marked matched data by colors)
encrypted_datatooliv
583C1A88BCFF2E025CEFEA1F15C9openssl tooldefault
953F04E5FDF7754EB89E39AAE12Fopenssl toolexplicit set 'iv'=0
302416d7806565d1a15e6b58666etools4noobsdefault (the only option here)
d27078f80c81d19e6b4a57154e33blowfish.online-domain-toolsdefault
3c38ba2e46c42a5538071fde89e3blowfish.online-domain-toolsexplicit set 'iv'=0
048a82b602dbe3a0387087643ce7blowfish-compat.online-domain-toolsdefault
3b3e546a514c8ca7980b0f379fc4blowfish-compat.online-domain-toolsexplicit set 'iv'=0
2cd10b1ad66f742c7db38fbab105bierkandt.orgdefault (the only option here)
94408e95ac0d6eac2aa3f9f61a80bierkandt.orgdefault (the only option here)
4c1e679fa0fcd8e4c8e45a411462bierkandt.orgdefault (the only option here)
c2252b7a79193bd512f1e4ff2476bierkandt.orgdefault (the only option here)
...

2e36059b6aa5482ffbe5fc3a7d0c3662sladex.org/blowfish.jsdefault (the only option here)
3c38ba2e46c42a5538071fde89e3python cryptoexplicit set 'iv'=0
3C38BA2E46C42A5538071FDE89E3php mcryptexplicit set 'iv'=0
3C49423A0F4F6550672974363137perl OpenSSL::Blowfish::CFB64explicit set 'iv'=0
3C49423A0F4F6550672974363137c/c++ OpenSSLexplicit set 'iv'=0

And. It looks like everybody has own version of blowfish cfb encryption. And only openssl & mcrypt gives the same results on different languages. Somebody even has own padding.
And everything is ok, till you don't need communicate between languages & libraries.
And also interesting part - openssl tool gives result, which differs from openssl lib. Let's see that.


lyrics

Okay, I don't want consider every tool - why every tool gives another result. But I will explain about why at least openssl tool and openssl library give different results.

This part I understood, when sit whole day in olly debugger and debugged openssl. Openssl - it's something like framework, where you can easily make crypto-chains of algorithms, and it seems a lot of work with connection parts happened during compilance. So, on static sources I couldn't understood how called blowfish from 'main' function. And here are some extracting of that experience.

First of all - here are exists key for openssl tool '-p':
> openssl.exe enc -p -bf-cfb -in file.txt -out file.bf -nosalt
(enter 'mypass' when it asked)
    key=A029D0DF84EB5549C641E04A9EF389E5
    iv =A10CE9C4682486F8

okay. let's repeat it:
    key=A029D0DF84EB5549C641E04A9EF389E5
    iv =A10CE9C4682486F8

okay. at least it stable) let's repeat and enter 'mypass2' as password
    key=29E80F01374C71764422B94532A4B336
    iv =836291E15B739ECD

And we see - not only 'key' != 'pass', here are also 'iv' produced by password.

So, the first big think to understanding - is: "Even if you are using -nosalt param - key isn't equal pass - key is md5(pass)"

Okay, let's try enter 'mypass' as key. Here are option '-K' special for that. Well, it accepts key as hex, so
    string 'mypass' in hex - "6D7970617373"
and
> openssl.exe enc -p -bf-cfb -in file.txt -out file.bf -nosalt -K 6D7970617373 -iv 0
    key=6D797061737300000000000000000000
    iv =0000000000000000
The point is - aligning of key. In console tool openssl key always aligned to 16 bytes, and olly debugger accepted these conclusions. Mcrypt & OpenSSL library - both works fine with any length of key.

In documentation (https://www.openssl.org/docs/apps/enc.html) the only phrase about that:
    "Blowfish and RC5 algorithms use a 128 bit key"

So:
    - key length always 16 bytes. If you have less - it completed it by zeros to 16 length.
    - key = md5(pass)
    - iv - derives from pass


problem solving in a short

  • use key '-p' to ensure you have key & iv values which you entered
  • key != pass. Without salt (option '-nosalt') key=md5(pass).
  • length of key for command line tool is ALWAYS 16 bytes
  • how 'iv' generated from pass - I don't know.

let's make friends - openssl tool & openssl lib

What do we know for now. To make friends openssl tool & openssl lib we need:
  • take key length 16 bytes
  • not mix up key & pass
  • add '-p' for ensuring we have write data
  • turn off salt
openssl tool    
> openssl.exe enc -bf-cfb -in file.txt -out file.bf -nosalt -K ABABABABABABABABABABABABABABABAB -iv 0 -p
key=ABABABABABABABABABABABABABABABAB
iv =0000000000000000
file.bf content: 63 C0 0D 29-6D F8 8D 4A-94 81 81 88-41 8C

openssl lib from perl
use Crypt::OpenSSL::Blowfish::CFB64;
my $crypt = Crypt::OpenSSL::Blowfish::CFB64->new("\xAB\xAB\xAB\xAB\xAB\xAB\xAB\xAB\xAB\xAB\xAB\xAB\xAB\xAB\xAB\xAB", pack( C8 => 0,0,0,0,0,0,0,0 ));
my $binary_data = $crypt->encrypt("my_lovely_text");
print $binary_data;

file.bf content: 63 C0 0D 29-6D F8 8D 4A-94 81 81 88-41 8C

So. At least here are clear - what to do openssl tool. Of course, we still have 'summary table', which shows that everybody has own understanding of blowfish cfb - but in the next note I will make mcrypt & openssl compatible.



conclusion

If you need encrypt/decrypt data only using openssl - then it's perfect.
If you need interact with something else - well, here are nuances, one of them - key_length always 16 bytes.


useful sources & links

actually so-called 'blowfish-compat' - it's only mcrypt thing and it's about endianing. [link]

add openssl modules to perl on windows

problem

Some perl cpan modules required openssl installed for your pc. Here are some of them:
  • Crypt::OpenSSL::AES
  • Crypt::OpenSSL::Bignum
  • Crypt::OpenSSL::Bignum::CTX
  • Crypt::OpenSSL::Blowfish
  • Crypt::OpenSSL::Blowfish::CFB64
  • Crypt::OpenSSL::CA
  • Crypt::OpenSSL::CA::AlphabetSoup
  • Crypt::OpenSSL::CA::Inline::C
  • Crypt::OpenSSL::CA::Resources
  • Crypt::OpenSSL::Cloner
  • Crypt::OpenSSL::Common
  • Crypt::OpenSSL::DSA
  • Crypt::OpenSSL::DSA::Signature
  • Crypt::OpenSSL::EC
  • Crypt::OpenSSL::ECDSA
  • Crypt::OpenSSL::PBKDF2
  • Crypt::OpenSSL::PKCS10
  • Crypt::OpenSSL::PKCS12
  • Crypt::OpenSSL::Random
  • Crypt::OpenSSL::RC4
  • Crypt::OpenSSL::RSA
  • Crypt::OpenSSL::SMIME
  • Crypt::OpenSSL::VerifyX509
  • Crypt::OpenSSL::X509
  • Crypt::SSLeay
That happens bcs they are just warppers over openssl functions.
When you are trying install such module - you get source code, which includes some header from openssl library & trying link with openssl library on your pc.
on unix-like systems this problem seems solves quickly - any of next commands:
  • sudo yum install perl-Crypt-SSLeay
  • sudo yum install openssl-devel
  • sudo apt-get install libssl-dev
that's so simple on unix-like systems because in unix you have some standard paths to headers & libraries. So, when you need to find library - you know where to search. On Windows, users don't have such cool thing - that's source of this problem.
And I didn't find any solutions of such problem in the Internet - so, solved & writed it here. Also I'm not pretend on some knowledge perl or it's package managers or something like that - I'm just using perl for solving my tasks, encounted problem and solved it.
In this note I will describe common principles of solving such problems - so, I think this experience can be expanded to any such problem.
to pass lyrics and just get solution - just scroll to "problem solving in a short".


tools & versions

I will describe on example of ActivePerl (and 64 and 32 bit version). Don't know what about strawberry perl, for example - but I think common idea will be the same.
I have ActivePerl v5.18.4
  • installed in C:\Perl64 for x86-64 version.
  • installed in C:\Perl for x86 version.
(I didn't install both of them in time - it must lead to confusion, but I've checked this approach on both versions - so, when you will see action for two versions - x86 and x86-64 - just choose your one)

Also you need to build openssl with mingw (for x86-64 target platforms - with mingw64).
How to do that - I described in note earlier - [link].
Why exactly mingw - bcs it's much simpler than add VisualStudio-compiled openssl. In the end of this note I will say how to add openssl, compiled in Visual Studio - but I don't like that.


Crypt::SSLeay

Module Crypt::SSLeay has easier solution that other mentioned, bcs it has some extra opportunities for installing. It has way to know where this libraries located ([link]):
> set OPENSSL_INCLUDE=...
> set OPENSSL_LIB=...
> cpanm Crypt::SSLeay

So, if you need only "Crypt::SSLeay" - you can just use it.
But this is specific only for this module - so, it's not universal method.


lyrics

What happened when you are trying install such module - for example 'cpan install Crypt::OpenSSL::Blowfish'? Well, first of all - if it's your fist 'cpan install' when you need compile stuff - it will bring you tools for compiling code (with mingw). You will see something like that:
--------------------------------------------------------------------------------------------------------
It looks like you don't have a C compiler and make utility installed.  Trying
to install dmake and the MinGW gcc compiler using the Perl Package Manager.
This may take a a few minutes...

Syncing site PPM database with .packlists...done
Downloading MinGW-4.6.3...done
Downloading dmake-4.11.20080107...done
Unpacking MinGW-4.6.3...done
Unpacking dmake-4.11.20080107...done
Generating HTML for MinGW-4.6.3...done
Generating HTML for dmake-4.11.20080107...done
Updating files in site area...
3697 files installed

--------------------------------------------------------------------------------------------------------
This utils will be located here:
    C:\Perl\cpan\build
or
    C:\Perl64\cpan\build

Then it will create directory in your cpan build directory
    C:\Perl\cpan\build
or
    C:\Perl64\cpan\build
in format:
    %package_name%-%version%-%six_looked_random_symbols%
(also it put .yml file with the same name, but for describing task it doesn't matter)

and here are extracted package archive and in this directory will be building process.
for example in 'Crypt-OpenSSL-RSA-0.28-gB_yPv'

here are located Makefile.PL file. For Crypt::OpenSSL this script uses module ExtUtils::MakeMaker. This 'Makefile.PL' script generates 'Makefile'. And here are the key thing - this script searches libraries and generates make file to linking with libraries, present on your pc!
When it can't find library it prints to err console stream messages like that:
...
Warning (mostly harmless): No library found for -lcrypto
Warning (mostly harmless): No library found for -lmoldname
...


Then, after Makefile.PL generated makefile, called 'dmake' util, which will try compile stuff accordingly to information in 'makefile' (on example installing Crypt::OpenSSL::Blowfish)
Blowfish.xs:5:30: fatal error: openssl/blowfish.h: No such file or directory
or stuff like that


So, the first step to solve this problem - copy your openssl includes directory to place where perl will find it:
    C:\Perl\site\lib\auto\MinGW\i686-w64-mingw32\include
or
    C:\Perl64\site\lib\auto\MinGW\x86_64-w64-mingw32\include

Then if you will try again 'cpan install Crypt::OpenSSL::Blowfish', it will find includes, but not find libraries:
Warning (mostly harmless): No library found for -lcrypto
...
Blowfish.o:Blowfish.c:(.text+0x125): undefined reference to `BF_encrypt'
Blowfish.o:Blowfish.c:(.text+0x151): undefined reference to `BF_decrypt'
Blowfish.o:Blowfish.c:(.text+0x2c3): undefined reference to `BF_set_key'


so, you need to put libraries where ExtUtils::MakeMaker will find it. Where ExtUtils::MakeMaker searches this files - you can understand by 'Process Monitor' tool [link]. Just turn on logging all events, run 'cpan install Crypt::OpenSSL::Blowfish', wait error message, save log to csv file and then grep by that. So, finally you will get list of locations where it searches libraries (part of log, grepped by 'ssl'):
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\-lssl32"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\libssl32.a"
        "C:\MinGW\x86_64-w64-mingw32\lib\libssl32.a"
        "C:\Perl64\lib\CORE\libssl32.a"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\libssl32.dll.a"
        "C:\MinGW\x86_64-w64-mingw32\lib\libssl32.dll.a"
        "C:\Perl64\lib\CORE\libssl32.dll.a"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\libssl32.lib"
        "C:\MinGW\x86_64-w64-mingw32\lib\libssl32.lib"
        "C:\Perl64\lib\CORE\libssl32.lib"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\ssl32.a"
        "C:\MinGW\x86_64-w64-mingw32\lib\ssl32.a"
        "C:\Perl64\lib\CORE\ssl32.a"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\ssl32.dll.a"
        "C:\MinGW\x86_64-w64-mingw32\lib\ssl32.dll.a"
        "C:\Perl64\lib\CORE\ssl32.dll.a"
        "C:\Perl64\cpan\build\Crypt-OpenSSL-RSA-0.28-XdSoEu\ssl32.lib"
as you can see - it searches libraries by names, which differs from that we got after compilcation. And also you can find "C:\Perl64\lib\CORE" ("C:\Perl64\lib\CORE" for x86-64) the best place to putting your libraries. I put it on these paths with these names:
    libssl.a    =>    C:\Perl64\lib\CORE\ssl32.a
    libcrypto.a    =>    C:\Perl64\lib\CORE\eay32.a
    libcrypto.a    =>    C:\Perl64\lib\CORE\libcrypto.a
+ copy this file (somewhy it searcher stuff like GetDeviceCaps):
    C:\Perl64\lib\CORE\libgdi32.a => C:\Perl64\lib\CORE\libgdi32.a
that's it.


dmake case

What if you have similar case, but it's more difficult?
For example - you have some dir with files in your C:\Perl\cpan\build or C:\Perl64\cpan\build directory - which stay here after unsuccessful installation of some cpan module. And you found way to fix makefile to build process finished correctly.
Then, to finish installation exec next commands:
> dmake
> dmake install



visual studio

Here are way to build with VisualStudio-compiled openssl.
First, when you put your lib files and try install some module, you will get:
        C:\Perl64\lib\CORE\libcrypto.a(tmp32/bf_cfb64.obj):(.text$mn+0x1e): undefined reference to `__chkstk'
        C:\Perl64\lib\CORE\libcrypto.a(tmp32/bf_skey.obj):(.text$mn+0x11): undefined reference to `__chkstk'
        C:\Perl64\lib\CORE\libcrypto.a(tmp32/bf_enc.obj):(.text$mn+0x1a): undefined reference to `__chkstk'
so, you need take
    C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\lib\chkstk.obj
for x86 or
    C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\lib\amd64\chkstk.obj
for x86-64.
put it somewhere in accessible place, for example
    C:\Perl64\lib\CORE\chkstk.a
and fix in makefile (in 'C:\Perl64\cpan\build'):

    LDLOADLIBS = C:\Perl64\lib\CORE\libcrypto.a C:\Perl64\lib\CORE\libgdi32.a
to
    LDLOADLIBS = C:\Perl64\lib\CORE\libcrypto.a C:\Perl64\lib\CORE\libgdi32.a C:\Perl64\lib\CORE\chkstk.a

then
> dmake
> dmake install
That's it. But I think way with mingw-compiled library is much simpler.


problem solving in a short

1. put includes from your mingw-compiled lib to:

    C:\Perl\site\lib\auto\MinGW\i686-w64-mingw32\include
or
    C:\Perl64\site\lib\auto\MinGW\x86_64-w64-mingw32\include

2. put your mingw-compiled lib to
    libssl.a    =>    C:\Perl64\lib\CORE\ssl32.a
    libcrypto.a    =>    C:\Perl64\lib\CORE\eay32.a
    libcrypto.a    =>    C:\Perl64\lib\CORE\libcrypto.a
    + copy
    C:\Perl64\lib\CORE\libgdi32.a => C:\Perl64\lib\CORE\libgdi32.a


useful sources & links
here are looks like another solution (but as I see - it's for strawberry perl) - [link]


APPENDIX - possible errors (for googling people):

main problem illustration - on example of 'cpan install Crypt::OpenSSL::SMIME'
------------------------------------------------------------------------------------------------------
Warning (mostly harmless): No library found for -lcrypto
Warning (mostly harmless): No library found for -lmoldname
Warning (mostly harmless): No library found for -lkernel32
Warning (mostly harmless): No library found for -luser32
Warning (mostly harmless): No library found for -lgdi32
Warning (mostly harmless): No library found for -lwinspool
Warning (mostly harmless): No library found for -lcomdlg32
Warning (mostly harmless): No library found for -ladvapi32
Warning (mostly harmless): No library found for -lshell32
Warning (mostly harmless): No library found for -lole32
Warning (mostly harmless): No library found for -loleaut32
Warning (mostly harmless): No library found for -lnetapi32
Warning (mostly harmless): No library found for -luuid
Warning (mostly harmless): No library found for -lws2_32
Warning (mostly harmless): No library found for -lmpr
Warning (mostly harmless): No library found for -lwinmm
Warning (mostly harmless): No library found for -lversion
Warning (mostly harmless): No library found for -lodbc32
Warning (mostly harmless): No library found for -lodbccp32
Warning (mostly harmless): No library found for -lcomctl32
Please specify prototyping behavior for SMIME.xs (see perlxs manual)
SMIME.xs: In function 'XS_Crypt__OpenSSL__SMIME_new':
SMIME.xs:232:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:233:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:234:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:235:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:236:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:237:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:238:3: warning: passing argument 3 of 'Perl_sv_2pv_flags' from incompatible pointer type [enabled by default]
C:\Perl64\lib\CORE/proto.h:3835:21: note: expected 'STRLEN * const' but argument is of type 'int *'
SMIME.xs:245:18: warning: assignment discards 'const' qualifier from pointer target type [enabled by default]
SMIME.o:SMIME.c:(.text+0x1a1): undefined reference to `BIO_new_file'
SMIME.o:SMIME.c:(.text+0x514): undefined reference to `X509_free'
SMIME.o:SMIME.c:(.text+0x520): undefined reference to `EVP_PKEY_free'
SMIME.o:SMIME.c:(.text+0x529): undefined reference to `X509_STORE_free'
SMIME.o:SMIME.c:(.text+0x535): undefined reference to `BIO_free'
SMIME.o:SMIME.c:(.text+0x61c): undefined reference to `X509_STORE_new'
SMIME.o:SMIME.c:(.text+0x62d): undefined reference to `X509_LOOKUP_file'
SMIME.o:SMIME.c:(.text+0x638): undefined reference to `X509_STORE_add_lookup'
SMIME.o:SMIME.c:(.text+0x660): undefined reference to `X509_LOOKUP_ctrl'
SMIME.o:SMIME.c:(.text+0x669): undefined reference to `X509_LOOKUP_hash_dir'
SMIME.o:SMIME.c:(.text+0x674): undefined reference to `X509_STORE_add_lookup'
SMIME.o:SMIME.c:(.text+0x693): undefined reference to `X509_LOOKUP_ctrl'
SMIME.o:SMIME.c:(.text+0x698): undefined reference to `ERR_clear_error'
SMIME.o:SMIME.c:(.text+0x6e7): undefined reference to `X509_STORE_free'
SMIME.o:SMIME.c:(.text+0x72e): undefined reference to `BIO_s_file'
SMIME.o:SMIME.c:(.text+0x736): undefined reference to `BIO_new'
SMIME.o:SMIME.c:(.text+0x754): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0x7b4): undefined reference to `BIO_free'
SMIME.o:SMIME.c:(.text+0x7d6): undefined reference to `d2i_PrivateKey_bio'
SMIME.o:SMIME.c:(.text+0x876): undefined reference to `d2i_PKCS12_bio'
SMIME.o:SMIME.c:(.text+0x895): undefined reference to `PKCS12_parse'
SMIME.o:SMIME.c:(.text+0x89d): undefined reference to `PKCS12_free'
SMIME.o:SMIME.c:(.text+0x905): undefined reference to `PEM_read_bio_PrivateKey'
SMIME.o:SMIME.c:(.text+0x938): undefined reference to `BIO_s_file'
SMIME.o:SMIME.c:(.text+0x940): undefined reference to `BIO_new'
SMIME.o:SMIME.c:(.text+0x967): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0x9d2): undefined reference to `BIO_free'
SMIME.o:SMIME.c:(.text+0x9ea): undefined reference to `d2i_PKCS12_bio'
SMIME.o:SMIME.c:(.text+0xa08): undefined reference to `PKCS12_parse'
SMIME.o:SMIME.c:(.text+0xa10): undefined reference to `PKCS12_free'
SMIME.o:SMIME.c:(.text+0xa4d): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0xa6c): undefined reference to `PEM_read_bio_X509_AUX'
SMIME.o:SMIME.c:(.text+0xd7b): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0xd9c): undefined reference to `PKCS7_sign'
SMIME.o:SMIME.c:(.text+0xdba): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0xdd5): undefined reference to `BIO_new_file'
SMIME.o:SMIME.c:(.text+0xdf7): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xe0d): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xe23): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xe41): undefined reference to `SMIME_write_PKCS7'
SMIME.o:SMIME.c:(.text+0xe4d): undefined reference to `PKCS7_free'
SMIME.o:SMIME.c:(.text+0xe52): undefined reference to `sk_new_null'
SMIME.o:SMIME.c:(.text+0xe70): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0xe9e): undefined reference to `sk_push'
SMIME.o:SMIME.c:(.text+0xeb2): undefined reference to `BIO_new_file'
SMIME.o:SMIME.c:(.text+0xee1): undefined reference to `PKCS7_encrypt'
SMIME.o:SMIME.c:(.text+0xeff): undefined reference to `BIO_ctrl'
SMIME.o:SMIME.c:(.text+0xf15): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xf2b): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xf41): undefined reference to `BIO_printf'
SMIME.o:SMIME.c:(.text+0xf5f): undefined reference to `SMIME_write_PKCS7'
SMIME.o:SMIME.c:(.text+0xf6b): undefined reference to `PKCS7_free'
SMIME.o:SMIME.c:(.text+0xf79): undefined reference to `X509_free'
SMIME.o:SMIME.c:(.text+0xf7e): undefined reference to `sk_pop_free'
SMIME.o:SMIME.c:(.text+0xf87): undefined reference to `X509_free'
SMIME.o:SMIME.c:(.text+0xf93): undefined reference to `BIO_free'
SMIME.o:SMIME.c:(.text+0xf9f): undefined reference to `BIO_free'
SMIME.o:SMIME.c:(.text+0x1678): undefined reference to `OPENSSL_add_all_algorithms_noconf'
SMIME.o:SMIME.c:(.text+0x167d): undefined reference to `EVP_des_ede3_cbc'
c:/perl64/site/lib/auto/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.6.3/../../../../x86_64-w64-mingw32/bin/ld.exe: SMIME.o: bad reloc address 0x0 in section `.pdata'
c:/perl64/site/lib/auto/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.6.3/../../../../x86_64-w64-mingw32/bin/ld.exe: final link failed: Invalid operation
collect2: ld returned 1 exit status
dmake.exe:  Error code 129, while making 'blib\arch\auto\Crypt\OpenSSL\SMIME\SMIME.dll'
------------------------------------------------------------------------------------------------------

what to do: make actions mentioned in this note.

problem, when includes found, and library not found.
------------------------------------------------------------------------------------------------------
CPAN: Module::CoreList loaded ok (v5.20141002)
cp Blowfish.pm blib\lib\Crypt\OpenSSL\Blowfish.pm
Running Mkbootstrap for Crypt::OpenSSL::Blowfish ()
C:\Perl64\bin\perl.exe -MExtUtils::Command -e chmod -- 644 Blowfish.bs
C:\Perl64\bin\perl.exe C:\Perl64\lib\ExtUtils\xsubpp  -typemap C:\Perl64\lib\ExtUtils\typemap  Blowfish.xs > Blowfis
h.xsc && C:\Perl64\bin\perl.exe -MExtUtils::Command -e mv -- Blowfish.xsc Blowfish.c
C:\Perl64\site\bin\gcc.exe -c           -s -O2 -DWIN32 -DWIN64 -DCONSERVATIVE -DPERL_TEXTMODE_SCRIPTS -DUSE_SITECUST
OMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -fno-strict-aliasing -mms-bitfields -s -O2         -D
VERSION=\"0.02\"        -DXS_VERSION=\"0.02\"  "-IC:\Perl64\lib\CORE"  -DPERL5 -DOPENSSL_NO_KRB5 Blowfish.c
Blowfish.xs:5:30: fatal error: openssl/blowfish.h: No such file or directory
compilation terminated.
dmake.exe:  Error code 129, while making 'Blowfish.o'
  VKRAMSKIH/Crypt-OpenSSL-Blowfish-0.02.tar.gz
  C:\Perl64\site\bin\dmake.exe -- NOT OK
CPAN: YAML::XS loaded ok (v0.52)
Running make test
  Can't test without successful make
Running make install
  Make had returned bad status, install seems impossible
------------------------------------------------------------------------------------------------------

what to do: make actions mentioned in this note - after putting includes in accessible place.