DPP: Linker issue with functions implemented in C header files

Tue Feb 18 09:20:08 UTC 2020

On Tuesday, 18 February 2020 at 08:32:47 UTC, Petar Kirov 
[ZombineDev] wrote:
> On Tuesday, 18 February 2020 at 05:41:38 UTC, Andre Pany wrote:
>> Hi,
>>
>> I try to get wrap the "Azure SDK for C" using DPP and have 
>> following issue.
>> Functions, which are actually implemented in C header files 
>> will cause
>> linker errors:
>>
>> https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/inc/az_span.h#L91
>>
>> Example:
>> AZ_NODISCARD AZ_INLINE az_span az_span_init(uint8_t * ptr, 
>> int32_t length, int32_t capacity) {
>>   return (az_span){ ._internal = { .ptr = ptr, .length = 
>> length, .capacity = capacity, }, };
>> }
>>
>> Error message:
>> /tmp/app.o:az_storage_blobs.d:function 
>> _D20blobs_client_example__T19AZ_SPAN_FROM_BUFFERTG4096hZQBdFNbQnZS16az_storage_blobs7az_span: error: undefined reference to 'az_span_init'
>>
>> I do not know C well, is this the expected behavior and should 
>> I ask the Azure SDK developers to not implement functions 
>> within C header files?
>>
>> Kind regards
>> André
>
> I think the problem is that you haven't actually linked in the 
> Azure SDK C library.
>
> Dpp translates the header declarations from C to D, but the 
> actual definitions (function bodies) are not part of the 
> process. The executable code for the function definitions 
> should be inside either a static or dynamic library provided by 
> the SDK.
>
>
> From the the project's readme file, it looks like they're using 
> CMake as the build system generator (afterwards both make and 
> ninja should be valid choices for building):
>
> mkdir build
> cd build
> cmake ../
> make
>
> In cases like this, it's best to checkout the CMakeLists.txt 
> files of the individual sub project, like this one:
>
> https://github.com/Azure/azure-sdk-for-c/blob/master/sdk/core/core/CMakeLists.txt
>
> As you can see, there are several outputs of the build process, 
> among which:
>
> - add_library(az_core ...)
> This defines a library named az_core which can produce either a 
> static (.a on Linux, .lib on Windows) or dynamic library file 
> (.so on Linux, .dll on Windows). (If no configuration is 
> specified, I think it's static by default).
> So the final file name would be libaz_core.{a,so} on Linux.
> For the .c files to be built, a list of include directories 
> must be specified, where the various .h would located 
> (containing function and type declarations). This done like so:
> target_include_directories (az_core PUBLIC ...)
> The 'PUBLIC' argument to the target_include_directories 
> specifies that if you want to use the library, you need to use 
> the same include directories, as those needed for building it.
>
> - add_executable(az_core_test ..)
> This defines an executable build output, which looks is only 
> used for testing, so it's not interesting to us, except that it 
> can serve as an example app using the az_core library.
>
> ---
>
> So in summary, if you want to use the az_core library, you need 
> to:
> 1. Build it
> 2. Run Dpp like so:
>
> d++ \
>   --include-path <specify the same as what was given to 
> target_include_directories>
>   <your D file(s) here)
>   <path to libaz_core.a>
>
>
> You will need to repeat the same steps for any other part of 
> the Azure C SDK.
>
>
> --------
>
> TL;DR
> After I went through all those steps I got a similar linker 
> error for az_http_response_init. After looking where is the 
> actual function definition, it turned out that it's not defined 
> in a .c file, but it is an inline function part of a header 
> file.
> Searching for az_span_init revealed the same (I could have 
> saved myself some time by reading your message more carefully 
> :D).
>
> So, to answer your original question, the problem is that dpp 
> translates only declarations, not function definitions (such as 
> inline functions like that).
>
> For now, your best course of action is to translate all inline 
> function definition by hand. Since in C inline functions are 
> mostly short and simple functions (a better alternative to 
> macros), hopefully that won't be too much work.
>
> Also, looking at macros like AZ_SPAN_FROM_STR, there's really 
> very little chance that they could be correctly translated 
> automatically. As the things they do are likely not valid even 
> in @system D code (without additional casts), so it's better to 
> write your own D functions by hand anyway.
>
>
> Here's what I tried:
>
> test.dpp:
>
> #include <az_http.h>
> #include <az_span.h>
>
> import std.stdio;
>
> void main()
> {
>     char[] resp =
>         "HTTP/1.2 404 We removed the\tpage!\r\n" ~
>         "\r\n" ~
>         "But there is somebody. :-)".dup;
>     az_span response_span =
>     {{
>         ptr: cast(ubyte*)resp.ptr,
>         length: cast(int)resp.length,
>         capacity: cast(int)resp.length
>     }};
>     az_http_response response;
>     az_result result = az_http_response_init(
>         &response, response_span);
>
>     writeln(result);
> }
>
> d++ --compiler ldmd2 --include-path ./inc test.dpp 
> ./build/libaz_core.a

Hi Petar,

thank you very much for the explanation and the code sample.
Filling the az_span anonymous member was the tricky part,
I thought it would be not possible to do so, but you showed me
the trick.

I will do it like you have proposed but had also already created
a ticket for the Azure SDK developer:
https://github.com/Azure/azure-sdk-for-c/issues/359
There should be a more convenient way to fill a az_span structure.

For reference, here is my dockerfile which does the DPP call and 
linking:

``` dockerfile
FROM dlang2/ldc-ubuntu:1.20.0 as ldc

RUN apt-get install -y git libssl-dev uuid-dev 
libcurl4-openssl-dev curl

RUN curl -OL 
https://cmake.org/files/v3.12/cmake-3.12.4-Linux-x86_64.sh \
     && mkdir /opt/cmake \
     && sh /cmake-3.12.4-Linux-x86_64.sh --prefix=/opt/cmake 
--skip-license \
     && ln -s /opt/cmake/bin/cmake /usr/local/bin/cmake

RUN git clone https://github.com/Azure/azure-sdk-for-c.git \
     && cd azure-sdk-for-c \
     && git submodule update --init --recursive

RUN cd azure-sdk-for-c \
     && mkdir build \
     && cd build \
     && cmake ../ \
     && make

RUN apt-get install -y clang-9 libclang-9-dev
RUN ln -s /usr/bin/clang-9 /usr/bin/clang
COPY az_storage_blobs.dpp /tmp/

RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- --help

RUN DFLAGS="-L=-L/usr/lib/llvm-9/lib/" dub run dpp -- 
/tmp/az_storage_blobs.dpp \
     --include-path /azure-sdk-for-c/sdk/core/core/inc \
     --include-path /azure-sdk-for-c/sdk/core/core/internal \
     --include-path /azure-sdk-for-c/sdk/storage/blobs/inc \
     --include-path 
/azure-sdk-for-c/sdk/transport_policies/curl/inc \
     --preprocess-only

ADD blobs_client_example.d /tmp/blobs_client_example.d
RUN  ldc2 /tmp/blobs_client_example.d /tmp/az_storage_blobs.d \
     /azure-sdk-for-c/build/sdk/core/core/libaz_core.a \

/azure-sdk-for-c/build/sdk/storage/blobs/libaz_storage_blobs.a \

/azure-sdk-for-c/build/sdk/transport_policies/curl/libaz_curl.a \
     -of=/tmp/app
```

Kind regards
André