D modules/sourcecode organisation/filesystem mapping

[FunkyM]

Getting to know D can roughly be compared to seeing light in the darkness. 

It appeared to me that the design of the language followed strict logic patterns
and enjoyable it solved cavejards of problems of other languages.

Next step was to evaluate it for use in commercial projects and spotting any
weakness it might have when starting to use it for a mid-sized project.

Surprisingly, it appeared to solve any assumed weak points itself, again
reflecting
it's really geniously designed semantics.

In affect, the D programming language seems as a good candidate to challenge the
upper mainstream competitiors and from what I see in the community there are
quite
a lot of people who forsee this aswell. It sure has a lot of potential (also
making
Walter Bright an "asked for" legend for businesses migrating to D in all areas).

Aside from the language design itself, first of all the overall "marketing" of
the language needs to "grow up" finally (intuitive web presence,
public repository for gdc, ...) in order to make more people aware of it.

Without a commercial backup, community efforts for this are already ongoing
(some in "secret" groups) to make this happen; some more, some less promising.

However, afterall I found an issue which I think will have an impact in this 
development of the language at some point, primary as soon as common big-sized
frameworks with extended API's are created using D (relates to Ares, Mango etc.
aswell).

It also has an impact on deciding to use D in some of my projects and the
decision to dedicate extended resources to help D's development/marketshare.

This most likely has not yet strived the round as it was not really a
requirement
for most small "libs" which act more like an utility than reaching a degree of 
complex frameworks and the current system was sufficient or basicly just taken
as-is.

# Issue:

D provides solutions to almost any "logic" weakness you could possibly find in
other languages and "counters" attempts to find them in D itself.

The D specification defines several goals and rules to base it's design on.
However, some of the rules fail especially on the main topic of my post:

>> D and module/sourcecode organisation/filesystem-to-module-mapping <<

# The "module" approach reveales the following weakness list:

- Not possible to have unlimited package hierarchies
(There is an "all.d" usage hack, D should practically NOT require "hacks"...)
- Porting bigger projects is not possible without introducing API changes
- Prevents creation of maintainable applications/frameworks/libraries
(req. for future of D!)
- Spltting up a bigger module file does break your API (see below)
- Adds additional unrequired package levels
(notice this in MANY places, e.g.: gtk.window.Window vs. gtk.Window)
- Bad maintainability if lots of classes are implemented in one module
- Ambigious "non-intuitive" API design rules for developers migrating to D
- Prevents splitting big sources into multiple files efficiently
- It only succeedes on old-fashioned "flat-hierarchy" C style "utility
libraries"
(Yes you can write big libraries; efficient/logical for a powerful OO language?)
- Limits/forces the developer indirectly to design "flat" libraries
(think of intuitive descriptive/naming needs in many cases)

These issues regarding mappings have been noticed on several occasions
while working with the D language primary in attempts to create application
frameworks such as the DFL GUI library, DUIT Gtk Library, x-2-D language
converters or similar libraries. In those cases either a big module file is
used or an additional duplicated package level has been introduced.

The authors of these WELCOME a proper solution if one would exist.

Some people are not even aware of these limitations yet but "intentionally"
propose module hierarchies that would not work due current limitations.

It is often an issue for projects in their initial design state or once they get
a lot of classes, alongside their implementation, in one single module file,
then
attempting to split them into individual modules.

Neither is it very efficient nor practicable from the programmers point of view.

# Why this is significantly important/What would it improve:

The argument set "modules are just simpler and better" is logically very weak if

compared to the other arguments used in all other areas of D's specification.

Modules do not solve the discussed issues nor stand to the modern "namespace"
element's functionality.

The requirement for D not to fail for serious complex development problems and
adapting modern approaches for maintable efficient API design is an requirement
for it's future ranking for being picked as the language for various projects
either for proprietary or open source projects with many developers.
(Imagine revision control systems on big source files)

Designing a cross-platform GUI library would suffer on these limitations.

Simple example:

-> Imaging you have such a structure:

system.drawing.SomeEnum.TOP   -> /system/drawing.d
system.drawing.Bitmap         -> /system/drawing.d
system.drawing.Brush          -> /system/drawing.d
..

-> At some point drawing.d becomes big/unmaintainable -> split sources

system.drawing.SomeEnum.TOP   -> CURRENTLY IMPOSSIBLE: packagename != modulename
system.drawing.bitmap.Bitmap  -> /system/drawing/bitmap.d with add. package lvl
system.drawing.brush.brush    -> /system/drawing/brush.d with add. package lvl
..

# No critic without... a potential solution:

After trying a couple of things it was hard to still be compliant to the D
goals/rules while providing a solution that is backwards compatible to current
D projects to avoid a need for re-write while still adding the required
functionality and not increasing compiler complexity significantly (such as the
namespace feature would).

Surprisingly, after various attempts, the solution resulted to be simple, yet
needs verifications from various other people on this list.

Especially after reading a post issuing that this is possible:
"foo.d: module bar; ..." "foobar.d: import foo; void main(..."

This example showed that the "internals" of the compiler would already be able
to compensate for the required changes solving the presented weakness.

# Proposal "Solution A":

Introduce/change rules:

1) Remove the limitation that a packagename can not be a modulename
2) Change definition of "module" keyword to "defines the package of the current
module"
3) The import statement qualified name maps to the filesystem
packages.modulename
(4) Optionally rename/change to a better keyword used in this context?)

This seems to pass logic and should be possible to implement
(as it seems to be indirectly supported already):

/gtk.d:
module gtk;

enum edges { LEFT, RIGHT, TOP, BOTTOM }
..

/gtk/window.d:
module gtk;

class Window {
..

mainapp.d:

import gtk;
import gtk.window; // Get's ./gtk/window.d and maps it into package gtk
..

As of now you could already get your gtk.Window qualification if you would just
put everything into the "gtk.d" module, the final result would be identical
to that of the example above.

The proposal, while still maintaining the current way of loading modules due to
imports, would make the class Window to be qualified as gtk.Window leaving a
nice API and allowing to structure your source in a maintainable way along a
couple of other improvements:

# Positive arguments:

- After a change, this remains backwards compatible to current libraries
(I am 100% sure all new projects will use this "natively" automatically and
existing projects will migrate asap)
- Everything still remains fully qualified and uniquely identificable
- Source maintainability is improved allowing splitting source code up into
files
- Overall change does not break with D's language semantics
(nor has direct impact on the language itself)
- Simplifies porting existing API's significantly
(1:1, helps a couple of existing projects!)
- Freedom to choose flat/hierarchic source structuring
- Still sticks to use of modules vs. "header/include files" or "nested
namespaces"
- Allows efficient and modern API design; without overhead package levels
- Does not break use of "build" tool (a commonly used tool to aid in
compilation)
- Spltting up a bigger module file at any time does not break your API

Of course it breaks clearly with the rules packagename!=modulename and with the
current use of the "module" keyword definition.

Both favoured to be dropped when compared to the extended functionality and 
simplification you get from doing so.

Mind that this is not a shoutout for heating up a "namespace" discussion as a
solution, as such stuff probably adds significant overhead to the compiler.

Neither I don't hope for getting "D is not Java/C#" arguments here as I am aware
of the design intentions of D; rather objective discussions on this topic.

This post is clearly requesting community VOTES for fixing this issue, possibly
but not necessarly using the presented solution.

I encourage anyone to propose a logical efficient solution apart from mine.
Even just sharing your thoughts if you ran into similar limitations yourself.

I am a fan of modern, standards compliant, logical and efficient approaches and 
very eager for the D language to succeed and get constructive comments on this.

For my part I do not see that D will "jump-over" to adulthood mid-/long-term
without a proper fix to the discussed issue and I don't see a better way of 
bringing this onto the table aside from this list.

Thanks for reading all this, I hope it makes sense! :)


--- Martin Szulecki