Wednesday, April 29, 2015

linking problem with C

Story


Recently I've got errors during building project (at linking step), spent a lot of time to solve it - wrote some perl script for solving my problem, but finally found simple and nice solution.
The problem occured when several .lib files stored the same symbol - Visual Studio in this case generates next message:
    1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: socket already defined in ws2_32.lib(WS2_32.dll)
that means:
    - library file Win32_Interop.lib contains symbol 'socket'
    - library file ws2_32.lib contains symbol 'socket'
and linker can't decide which instance must be connected when somewhere called socket().

Examples of such errors (which I met):
    - yara defines symbols (strlcpy,strlcat) which is also defined in mysqlclient.lib
    - redis defines symbols (pthread_create, pthread_cond_init, pthread_cond_destroy, pthread_cond_wait, pthread_cond_signal) which is also defined in mysqlclient.lib
    - redis defines symbols (socket, WSASend, WSARecv, WSACleanup, ioctlsocket, setsockopt, getsockopt, connect, listen, bind, shutdown, htons, htonl, getpeername, getsockname, ntohs, select, ntohl, freeaddrinfo, getaddrinfo, WSASetLastError, WSAGetLastError, WSAIoctl) which is also defined in standard windows library ws2_32.lib

There are simple way - say to linker smthng like "choose them in random way":
'properties -> Linker -> General -> Force File Output' with any of next values:
    "Enabled (/FORCE)"
    "Multiply Defined Symbol Only (/FORCE:MULTIPLE)"
but it can lead to some mystic bugs. So, I don't like that way.

Here are facts which obviously lead to solution:
    - All such disambiguated symbols which I saw - it's only for internal usage in libraries. No one such symbol designed for calling outside.
    - And all such symbols I saw - was defined in one file, and called from another - so, I couldn't just make them 'static'.
    - All these projects I'm building from source code.
And here are obvious solution - if it's internal library stuff - who cares which names these internal symbols have.

So, task became 'replace all instances of lexem in projects to another lexem'.
It's refactoring task - and seems I can use typical refactoring tools. but here are things:
    - paid software (klocwork, Slickedit) - I need to use it only once to create lib file. I don't want to pay for that. And trial sucks.
    - some plugins for some IDE (Eclipse CDT, Xrefactory for Emacs, Netbeans, Qt Creator) - I don't think they have batch mode. I don't want manually click for refactoring. And I don't want install some huge ide.
    - GCC MELT - maybe, but I don't want gcc.
Actually one tool looked nice - cscope - but on the first meeting it looked creepy. And I still didn't find out how it works.

I searched ways to solve it by myself.
Simple s/lexem_A/lexem_B/ obviously won't work because if you want replace WSACleanup in redis to any another lexem - it ruins next constructions:
    auto f_WSACleanup = dllfunctor_stdcall<int>("ws2_32.dll", "WSACleanup");
because string constant will be replaced also.
Also if lexem_A is substring of some another lexem - it also will be affected.

Make lexical analysis is difficult. Especially if you are newbie in this area.

On that time I wrote perl script, which uses some simple heuristic analysis of context. Tested it on 2 projects - it worked fine. But here are simple & nice solution.


Simple and nice solution

Answet is preprocessing.
C/C++ compiler apply defines to lexems - it won't affect on substrings of another lexems; it won't affect on string constants. Ideal solution.
Even more - Visual Studio allows set defines without modifications of any source code. And even more - Visual Studio allows you do it with several projects in time! You can set defines for all projects in solution.
Just select c/c++ projects by clicking on projects with pressed 'ctrl' button (if you select also c# project, for example - nothing will work. Only c/c++ projects must be selected).
Then click right mouse button and go to
    "Properties" -> "Configuration Properties" -> "C/C++" -> "Preprocessor" -> "Preprocessor Definitions"
and add what do you want to replace in format lexemA=lexemB. For example - here are my stuff for yara (to not conflict with mysqlclient.lib):
strlcpy=libyara_internal_strlcpy
strlcat=libyara_internal_strlcat


That's it. Apply it & compile - all lexems will be replaced.


Examples

--------------------------------------------------------
redis (don't forget you don't need select c# project - "ReleasePackagingTool"):
pipe=redis_internal_pipe
socket=redis_internal_socket
fdapi_close=redis_internal_fdapi_close
open=redis_internal_open
setsockopt=redis_internal_setsockopt
fcntl=redis_internal_fcntl
poll=redis_internal_poll
getsockopt=redis_internal_getsockopt
connect=redis_internal_connect
read=redis_internal_read
write=redis_internal_write
fsync=redis_internal_fsync
fdapi_fstat64=redis_internal_fdapi_fstat64
listen=redis_internal_listen
ftruncate=redis_internal_ftruncate
bind=redis_internal_bind
shutdown=redis_internal_shutdown
htons=redis_internal_htons
htonl=redis_internal_htonl
getpeername=redis_internal_getpeername
getsockname=redis_internal_getsockname
ntohs=redis_internal_ntohs
ioctlsocket=redis_internal_ioctlsocket
inet_addr=redis_internal_inet_addr
gethostbyname=redis_internal_gethostbyname
inet_ntoa=redis_internal_inet_ntoa
fdapi_fwrite=redis_internal_fdapi_fwrite
fdapi_setmode=redis_internal_fdapi_setmode
WSASetLastError=redis_internal_WSASetLastError
WSAGetLastError=redis_internal_WSAGetLastError
WSAIoctl=redis_internal_WSAIoctl
WSASend=redis_internal_WSASend
WSARecv=redis_internal_WSARecv
WSACleanup=redis_internal_WSACleanup
WSAGetOverlappedResult=redis_internal_WSAGetOverlappedResult
WSADuplicateSocket=redis_internal_WSADuplicateSocket
WSASocket=redis_internal_WSASocket
select=redis_internal_select
ntohl=redis_internal_ntohl
isatty=redis_internal_isatty
access=redis_internal_access
lseek64=redis_internal_lseek64
fdapi_get_osfhandle=redis_internal_fdapi_get_osfhandle
fdapi_open_osfhandle=redis_internal_fdapi_open_osfhandle
freeaddrinfo=redis_internal_freeaddrinfo
getaddrinfo=redis_internal_getaddrinfo
inet_ntop=redis_internal_inet_ntop
accept=redis_internal_accept
pthread_self=redis_internal_pthread_self
pthread_create=redis_internal_pthread_create
pthread_cond_init=redis_internal_pthread_cond_init
pthread_cond_destroy=redis_internal_pthread_cond_destroy
pthread_cond_wait=redis_internal_pthread_cond_wait
pthread_cond_signal=redis_internal_pthread_cond_signal

--------------------------------------------------------
yara:
strlcpy=libyara_internal_strlcpy
strlcat=libyara_internal_strlcat

--------------------------------------------------------


Script

Just put it here: https://github.com/HostageBrain/my_lovely_refactor
It analyzes context and replaces lexems which you needed.
Usage:
    - you are specifying extensions
    - you are specifying lexems for replacing
    - you can optional add custom data types for heuristic (redis required 2 such items)
    - you are running 'my_lovely_refactor.pl path_to_directory'
And it prints files where replacings happened. Original file saved in .N.bak extension, where N - nearest free number.


Errors for googling people

if you are trying compile together redis, mysql & ws2_32
--------------------------------------------------------
1>------ Build started: Project: blablabla, Configuration: Debug x64 ------
1>Win32_Interop.lib(win32fixes.obj) : error LNK2005: pthread_create already defined in mysqlclient.lib(my_winthread.obj)
1>Win32_Interop.lib(win32fixes.obj) : error LNK2005: pthread_cond_init already defined in mysqlclient.lib(my_wincond.obj)
1>Win32_Interop.lib(win32fixes.obj) : error LNK2005: pthread_cond_destroy already defined in mysqlclient.lib(my_wincond.obj)
1>Win32_Interop.lib(win32fixes.obj) : error LNK2005: pthread_cond_wait already defined in mysqlclient.lib(my_wincond.obj)
1>Win32_Interop.lib(win32fixes.obj) : error LNK2005: pthread_cond_signal already defined in mysqlclient.lib(my_wincond.obj)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: socket already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSASend already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSARecv already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSACleanup already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: ioctlsocket already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: setsockopt already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: getsockopt already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: connect already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: listen already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: bind already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: shutdown already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: htons already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: htonl already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: getpeername already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: getsockname already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: ntohs already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: select already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: ntohl already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: freeaddrinfo already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: getaddrinfo already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSASetLastError already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSAGetLastError already defined in ws2_32.lib(WS2_32.dll)
1>Win32_Interop.lib(Win32_FDAPI.obj) : error LNK2005: WSAIoctl already defined in ws2_32.lib(WS2_32.dll)
1>D:\projects\blablabla\blablabla.exe : fatal error LNK1169: one or more multiply defined symbols found
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========


--------------------------------------------------------
what to do - use defines (read note from the beginning)


if you are trying compile together mysql C connector & yara
--------------------------------------------------------
1>------ Build started: Project: blablabla, Configuration: Debug x64 ------
1>libyara64.lib(strutils.obj) : error LNK2005: "unsigned __int64 __cdecl strlcat(char *,char const *,unsigned __int64)" (?strlcat@@YA_KPEADPEBD_K@Z) already defined in mysqlclient.lib(crypt_genhash_impl.obj)
1>D:\projects\blablabla\blablabla.exe : fatal error LNK1169: one or more multiply defined symbols found
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========


--------------------------------------------------------
what to do - use defines (read note from the beginning)

1 comment: