DMD Source Archive - Why?

[Steven Schveighoffer]

On Thursday, 11 April 2024 at 01:36:57 UTC, Walter Bright wrote:
> We certainly could do more with .sar files, we just have to 
> start somewhere.
>
> If we're going to add features to a .tar file, like an index, 
> aren't we then creating our own format and won't be able to use 
> existing .tar programs?

No. tar programs would work fine with it. We could indicate they 
are normal files, and normal tar programs would just extract an 
"index" file when expanding, or we could indicate they are 
vendor-specific extensions, which should be ignored or processed 
as normal files by other tar programs. We are not the first ones 
to think of these things, it is in the spec.

> Yes, one can skip through a .tar archive indexing as one goes. 
> The problem is one winds up reading the .tar archive. With the 
> .sar format, the index is at the beginning and none of the rest 
> of the file is read in, unless actually needed. .tar is the 
> only archive format I'm aware of that does not have an index 
> section, and that's because it's designed for append-only 
> magtapes. (Talk about ancient obsolete technology!)

This would be a fallback, when an index isn't provided as the 
first file. So normal tar source files could be supported.

> Many archive formats also include optional compression, and 
> various compression methods at that. All that support would 
> have to be added to the compiler, as otherwise I'll get the bug 
> reports "dmd failed with my .zip file!"

tar format doesn't have compression, though the tar executable 
supports it. I wouldn't recommend zip files as a supported 
archive format, and using compressed tarballs would definitely 
result in reading the whole file (you can't skip N bytes when you 
don't know the compressed size).

> Still, the concept of presenting things as a single file is 
> completely distinct from the file format used. The archive 
> format being pluggable is certainly an option.

I stress again, we should not introduce esoteric formats that are 
mostly equivalent to existing formats without a good reason. The 
first option should be to use existing formats, seeing if we can 
fit our use case into them. If that is impossible or prevents 
certain features, then we can consider using a new format. It 
should be a high bar to add new file formats to the toolchain, as 
this affects all tools that people depend on and use.

Think of why we use standard object formats instead of our own 
format (which would allow much more tight integration with the 
language).

-Steve