Big problem with Small programs

[Sean Kelly]

Sean Kelly wrote:
> Walter Bright wrote:
>> Sean Kelly wrote:
>>> The first block goes from line 410 to line 6719 in the map file, and 
>>> the second block goes from line 8047 to line 14559 in the map file.  
>>> All told, that's 12821 separate constants or initializers in the 
>>> manually linked version that seem absent from the library-based version.
>>
>> That means that, in the library based version, nothing in the 
>> explicitly linked .obj files referenced them.
> 
> If that's true then nothing in either version referenced them.  The 
> application is identical in both cases but for some of the objects 
> living in a library in the small case.

Just some further discussion.  After exhaustive experimentation 
yesterday with Tango the following seems to be true:

Using the Bud/Build approach of compiling all modules to object files 
and then linking them always works, however on Win32 it may result in a 
surplus of data in the executable.  This may actually be legitimate 
behavior based on how the linker works and how the OMF blocks are 
segmented in the object files at link time, but it seems odd.

Using the library approach of compiling all library code into a static 
library and then linking it does not always work, but it results in much 
leaner executables on Win32.  Little or no surplus data is added to the 
executable, including data that is added using the Bud/Build method, 
even for the exact same application.  This may again be legitimate 
behavior based on how the lib tool splits/joins/moves OMF blocks when 
assembling the library (assuming it does so), or it may have something 
to do with the linker having "all the information at its fingertips" 
when assembling the application.

Where the library approach fails is if the library contains template 
code.  Results are inconsistent, but yesterday we were seeing link 
errors in some cases and GPFs in others just for different library 
modules using the same template code, all of which work perfectly using 
the Bud/Build method.  For the link errors, as near as we could tell the 
object files contained all the relevant functions (many of the link 
errors referred to class TypeInfo instances) but for some reason they 
didn't appear to be available at link time.  It is possible that the lib 
tool is throwing them out for some reason, but I could not speculate on why.

This leaves us in an awkward situation for releasing Tango.  Suggesting 
everyone use Bud/Build seems the most reliable but some may not be happy 
with large executables or with having to use Bud/Build at all. 
Assembling everything into a single static library seems unreliable at 
best, but it will undoubtedly be preferred by many people.  At the 
moment, I suspect we will adopt a hybrid approach out of necessity, 
placing some of the larger packages (like the Win32 package) in static 
libraries and linking them either explicitly or implicitly via 
pragma(lib) (which is problematic in itself since it doesn't work if 
placed in include files, at least with DMD--Bud may be different, I have 
yet to test this and plan to do so today).

I am hoping that we can produce a small test case for the template 
issues so we can move to a full library approach once the problems have 
been addressed.  Alternately, perhaps we can find a way to address 
linking so that less unnecessary data is included while linking objects 
directly.  I'll admit that at this point I'm of half a mind to simply 
write an OMF linker that takes as long as necessary to produce a working 
application and be done with it.

Out of curiosity, I suppose it would be a substantial amount of work for 
DMD (and I suppose DMC) to produce COFF object files instead of OMF? 
And I don't suppose that some work in this area may be necessary for 
generating 64-bit executables?  If there's a chance we could move to a 
newer object file format for Win32, we would at least gain access to a 
broader array of tools, even if COFF itself is a bit problematic.


Sean