Kotlin Meta and CT programming vs D

[Jo Blow]

On Monday, 23 December 2024 at 12:02:47 UTC, Renato Athaydes 
wrote:
> On Thursday, 19 December 2024 at 06:04:33 UTC, Jo Blow wrote:
>> But there have been several cases when I have asked others 
>> about trying to do certain "compile time" things in Kotlin 
>> that say it can't be done. It seems that there is actually no 
>> compile time programming in Kotlin(and I guess Java too?)? 
>> That at best one has to work in some DSL like thing that uses 
>> a "plugin" but things that make D shine because you can just 
>> get shit done, usually several ways, doesn't exist. Things 
>> like string mixins, CTFE, etc. D would be an amazing language 
>> to use if it was integrated into android studio like Kotlin.
>
> Those features are exactly what makes D such a difficult 
> language to provide good support for in an IDE. Even things 
> that are trivial in languages like Kotlin, like removing unused 
> imports, can't be easily done in D (I was told that in a 
> discussion about why that didn't work at all with D).

I don't think that is the case. Even with Kotlin I'm constantly 
getting import issues with auto import where it will import some 
random type is I copy and paste something. The removal part is 
essentially easy because you could just remove them all and then 
re-add if your adder was perfect(slow but).

Basically it's just a big import table graph. I think what makes 
it harder on D's end is lack of IDE support and the ad hoc nature 
of organizing libraries. In Koltin's ecosystem everything seems 
to work together and evolve in harmony while in D it's all add 
hock as people just tacked on stuff that they wanted and then it 
they eventually leave and others try to maintain it and so on. 
It's the difference of having a lot of $$$ behind something 
verses relying on "handouts".

There is literally nothing to stop D or any other language from 
having anything else any other does but time/$$$.

The underlying system is just mathematics and of that is just 
logic and all that stuff is perfectly well worked out.

Maybe one of the best languages might be  Haskell as essentially 
it is an implement of "category theory" but it has the least 
support. Likely the more "complex" a system is the less users it 
will have and so the less $$$ behind it. This is why python is so 
popular but a terrible language(it doesn't need to exist). It was 
pushed as a simple language of simple programmers(mostly who were 
just entering the boom of programming in the 2000's) and a lot of 
$$$ got behind it but it's offers nothing of value as a 
programming language. OTH it is easy to use and works and has a 
lot of libraries(because of the user base) so in that sense it 
does offer something and something which D doesn't have.

I think the way things have evolved are perceived wrongly. E.g., 
python is no more complex or simple than C. They are essentially 
the same languages with just different rewrite rules and extra 
features or not. But The transformation of society into a digital 
one and C being before that(and things like Fortran, Algo, and 
other before or around that) was considered "hard" because, well, 
it was in a different era. As people tried to learn computers and 
programming it was difficult(no wikipedia, intellisense, nice 
GUI's, speed, etc) so these languages became "hard languages". As 
society started to change and more $$$ was pushed in we get newer 
languages and some of these were pushed as "easier" and it hooked 
a lot of people. But, in fact, they are no easier or harder but 
just look different. It's like driving two cars. Pretty much if 
you can drive a car you can drive any car. You might have some 
trouble if you have to drive a manual and haven't learned that 
but it's not hard but just a little work. Similarly, you could 
say that manuals were first and then autos came out and it turned 
manuals into being "hard" when before they were just what they 
were(neither hard or easy but there wasn't anything to "divide 
and conquer" them).

It's basically all about $$$. with enough money anything can be 
done because one can use the good ol capitalism to get people to 
do the work.

I don't think anyone wants to spend a huge amount of money 
writing a new system that is much better designed from the ground 
up because the cost vs reward is too great. The productivity, 
say, might increase 10% but is that really worth the 
cost(specially in our current world environment). It likely is 
much cheaper to just hire more "slaves" to overcome that 10%.






> Kotlin and Java have always taken IDE integration seriously and 
> would not add a feature that "breaks" the IDE, even if that 
> makes compile-time programming impossible. Reflection is 
> heavily used by frameworks, but nearly never in application 
> code, exactly because it makes IDE support and static analysis 
> impossible.

Yes, impossible... because one can't know something that is 
unknowable and reflection typically isn't knowable at compile 
time. Yet, there are things that are known at compile time and 
must be known.

E.g., the type of any variable must be known at compile time. 
Even if it is "dynamic" it must be known that it is of type 
dynamic. Hence reflecting over the type is a well defined. There 
are many things that are defined at compile time that can be 
reflected over. Some types may be provably known to be something 
or variables may be specific variables(constants) and all of this 
can be used if it is tracked by the compiler.

I think the real issue here is not so much that reflection is 
"bad"... although it might be slower, but that there are 
generally ways that are used to overcome the need to reflect on 
types in application code.

The issue is that originally the type system in languages were 
very limited and fixed in nature. Compilers were very simple and 
"dumb". Things needed to be explicitly done. As things evolved 
things evolved. People learned a little more and added a little 
more.

I think the real problem is that we've been building on systems 
that were build on very primitive ideas and so it's a constantly 
piling on of ad-hoc changes that may or may not evolve into 
something more and then a constant need to maintain them.

Ideally if, say, we could just create a new system from the 
ground up(including hardware) with all the "learned lessons" we 
would get systems that would be far more efficient and "complex". 
It is because we can, as a species, handle the complexity more. 
But what we really have is having to work with essentially a 
primitive system with many layers upon layers of improvements to 
give it the features and expressiveness we really want from 
realizing there are better ways.

The issue is that the cost to truly start fresh is too much to go 
back and "do things right" with the lessons learned. So we are 
stuck with the flaws of the past that has become part of our 
"DNA". [This is true of all things because it is evolution at 
work]

The point I'm making is that none of this is "impossible" but 
simply impractical due to the cost involved.

The mathematics is pretty much completely worked out. Computers 
are just calculators and how they work are dictated by relatively 
straightforward mathematical ideas.

When we program we are pretty much just doing high level semantic 
bit fiddling. There's only one "god" and that is nand or nor. All 
logic, all math, all science, is just nand or nor and binary 
input and output. (it's all transistors at the end of the day)

So all this "stuff" we have in programming and computers is just 
abstractions to bridge the gap of how our minds like to represent 
things(very high level) vs the hardware layer of bits.

I guess we'll have to wait until some new discovery happens that 
kicks off a new path which will enable starting fresh in many 
ways. (the life and death are the same thing)

Maybe this will happen if quantum computing actually ever does 
anything. [Of course it will evolve long the same lines in the 
sense that it will start ad-hoc and build into itself alot of new 
types of flaws but it will have less of the old flaws]

>
>> Anyways, the entire point of this point is to ask this 
>> question: Is anyone who is pretty familiar with Kotlin and 
>> it's compile time programming willing to explain what the heck 
>> is really going on with it and why it seems so different to 
>> D's?
>
> As someone coming the opposite way, D metaprogramming looks 
> like pure magic. It's mind blowing (and so completely 
> underrated, Zig is getting all the attention for having 
> something similar but still less powerful). But yeah, the cost 
> of those features seem to be bad IDE support, which kind of 
> sucks, for sure.

To me D is great and it doesn't seem like magic at all. It was 
what I was looking for. Before D I was doing C# and so I guess it 
is appropo since D comes after C# in the "musical scale"  ;) In 
other languages I was using I would always want to do certain 
things to avoid the repetativeness and to find elegant and robust 
solutions.

I have programmed in assembly quite a bit when I was younger as 
well as C and C++ and many other languages(lua, JS, VBS, pascal, 
basic, php, perl, etc) so I got used to the idea that all 
languages are effectively the same with just different "vocab". 
Similar to natural languages that essentially are all the same 
even though they might have significant difference such as how 
they arrange the grammatical elements it is really just a 
"permutation" and "concatenation" issue. The idea is the same 
under the hood of all of them(after all, they all get translated 
into binary).

So it was always about the "features" the language had that could 
reduce the "visual complexity" or increase the expressiveness. 
E.g., imagine a language without loops where you had manually add 
each iteration as a new block and check. The entire point of a 
for loop is to abstract over this common "pattern". Just think 
how slow programming would be if you had to do it 
"manually"(another reason I don't like driving manuals. Not 
because I can't but because it's time consuming and overs nothing 
for me). Also, there were programming languages that did not have 
for loop. Most hand held calculators do not have for loops(or 
only for certain operations).

Now, if you understand that then that pretty much is ALL of 
programming in a nut-shell. It's all about finding better ways to 
do things(usually common things as to benefit through abstracting 
out the repetition).


Also, I also studied quite a bit of mathematics so I was used to 
this idea of abstractions. E.g., think of algebra and using 
variables. Same idea as programing with variables. If you think 
of arithmetic then that is sorta like programming without "meta 
programming". Your stuck doing everything by hand and things get 
very tedious.

When you go to algebra you can start doing things that would be 
very complicated in arithmetic very quickly. That is the 
abstraction process.

What is algebra? It is working with types. Arithmetic is working 
with the elements of the type. Arithmetic is when you are forced 
to use the "built in types" and can't use your own. You are stuck 
doing calculations on those types(e.g., integers or rationals).

Algebra starts to let you have some "wiggle room" with types. 
Usually you are still thinking in terms of fixed types in 
arithmetic(numbers) but it starts to get you thinking away from 
such things just because it is more abstract.

But algebra typically is quite limiting still to numbers. You 
have variables(rather than everything being constants) but it's 
power is only in abstracting out numbers.

Well, the point is that you can continue going. E.g., what if 
types themselves were variables? What if we could do "arithmetic" 
on types(rather than just numbers).

When you start going down this path you realize that you have a 
common pattern of basically types of ... of types. It's types all 
the way down. Everything is a type. Even 3 is it's own type.

Category theory embodies this concept by forcing us to think in 
terms of relationships between types and what is the correct way 
to think about them(using things like composition, functors(how 
types relate to each other), natural 
transformations(relationships between relations), etc.

So when you program enough and do math all this stuff just falls 
out. It's not something really explainable because it has to be 
experienced but you also eventually have to learn the formal 
"mathematics" to realize that people have already developed this 
stuff to a very high degree.

So at some point "D becomes inevitable". At some point algebra 
becomes inevitable and then abstract algebra and then category 
theory. It's just evolution at work.

The more you work in D the more working in the higher type 
levels(types of types) becomes natural. Just as the more you do 
algebra the more it becomes natural and arithmetic gets put on 
the back burner and becomes easy.

Then you want to work in types of types of types but even D 
becomes hard to use at this level. You start realizing that D is 
limited and start to look for something better. Then you come 
across Haskell and realize it can do all of what D could, in 
terms of the type system, and is even better.

With programming, though, at some point it becomes pointless to 
keep going up the type system.a the "reward" for using higher 
level abstracts becomes minimal vs just using what you got. It's 
also personal. Some people are ok with wasting time unrolling a 
loop for no reason what so ever but just because they can.

Category theory is the sort of "ultimate programming language" in 
that it has the tools to work at any level of abstraction(so 
types^5324325 is possible and the abstract looks exactly the same 
as types). So it sort of melds all the complexity of "meta 
programming" into just programming. That is why it's math of 
course. Haskell attempts to approximate this but it really is 
just stuck at the same level as D or slightly higher but with a 
more "type friendly" syntax.

But the point here is that it's all about abstraction. There is 
nothing more to it. But abstraction without realization is 
meaningless and so for computers there has to be some way to 
compose all that abstraction up into bits that connect to 
transistors that do the thing the abstraction is suppose to 
represent. This means that if there is any point in the the 
process of "concreatization" of an abstraction that can't be 
resolved to something known at compile time or knowingly 
determined at runtime one can't resolve it to just bits. There 
will be some spots where no bits exist but which all bits need to 
be known to exist.

Basically if you think of a compiler as "flattening" all the 
abstractions you have to actually be able to flatten them and 
that might not be the case if abstractions are not properly 
formed/well-defined.

Hence programming languages offer various degree's of "meta 
programming" of the "type system" and the more you learn to think 
abstractly the more you'll run into their limitations and desire 
more from it. Sometimes even very simple metaprogramming tasks 
are not implemented and this can be a PITA for someone who is 
used to using them. It becomes very difficult to go back down the 
ladder.





> Kotlin has annotation processors, like Java, which are about 
> the only metaprogramming you can do outside of just writing 
> code that explicitly generates source code (which you can do 
> easily enough with https://github.com/palantir/javapoet for 
> Java, or the Kotlin spinoff: 
> https://square.github.io/kotlinpoet/
>
> This is the current Kotlin solution: 
> https://kotlinlang.org/docs/ksp-overview.html
>
> As far as I know, almost no one uses this though. People are 
> happy to just NOT do anything at compile-time, I guess it's 
> hard to believe that if you come from D, but it's indeed 
> possible.

Yes, this is the strange thing I still struggle with. I hear a 
lot of talk about "compiler plugins" and "annotations" and "KSP", 
"KAPT', etc but none of this stuff is what I would really call 
meta programming or it is very basic and extremely complex for 
doing things that should be extremely simple.

E.g., in D I can do mixin("int x;"); instead of int x;

This might seem moronic to do but I can also do mixin("int 
"~name~";"); which is is like kotlins string interpolations but 
this literally creates "variable code".

Now I can generate code that depends on other code(whatever name 
is) and as long as the compiler can resolve what name is at the 
type of compilation(not runtime) then it can evaluate that string 
to a constant and hence insert it into the code *as if* I typed 
it by hand.

It's beautiful and very simple(the syntax might not be pretty but 
it's simple and so powerful and so it is beautiful in the meta 
programming sense).

That simple idea can be extremely powerful and used to reduce 
very complex things when used with other similar metaprogramming 
ideas. It allows you to do all kinds of things. You can parse 
strings using regex at compile time.  You might convert a text 
file(read at compile type) into D.

Basically you turn the compiler itself into an application or 
utility and it is all integrated into the language.

It solves about 95% of meta programming that one would ever want 
to do. The only thing I have found that D does not do well is 
higher level meta programming concepts like functors and natural 
transformations(and hence adjunctions, kan extensions, etc).

E.g., if I want to transform an entire class hierarchy into 
another one it could be very complicated but if D understood 
these higher level ideas it could be just one line of code. This 
is where Haskell does a much better job. Of course one can 
parameterize the class type and this gives one some power but it 
is "linear" in the sense that the transformation doesn't really 
do anything advanced. So what happens is every once in a while, 
when you know about these things, you'll say "I want to do this" 
and you just can't do it in D or at least not in any way that is 
desirable but you know there is a better way to do it. These 
typically happen when you want to mess with "large abstractions" 
or "abstracts of abstractions". That is why D really is really 
just a language with a single level of meta programming while 
Haskell is several levels(it's not easy to define this stuff 
though because sometimes it might have more or less depending on 
the specifics).





>
>> The way I see D's metal language was that essentially anything 
>> that was "defined at compile time" could be treated as a 
>> program at compile time and all the compile time information 
>> would be filled in as such since it could. If code could be 
>> determined to be completely well-defined at compilation then 
>> it could simply be "executed"(evaluated) at compile time(in a 
>> first pass). Kotlin doesn't seem to have anything like this. 
>> It doesn't even have string mixins or eval without including a 
>> kotlin compiler inside the language.
>
> Correct. Kotlin and Java people would be horrified of those 
> features, I think :D
>
> Again, because of IDE and tooling support.
>

well, Visual D handles them just well. I guess you mean that the 
IDE won't be able to provide various helper info? I haven't used 
android studio enough to know if it offers advanced things but it 
really doesn't do any more than what visual D does. It does do it 
better but I've not seen anything but basically syntax checking, 
intellisense, and various helper things like "import" or lint 
stuff or whatever. Only intellisense really matters a lot to me. 
I like that it can show some help in a window and it's 
refactoring stuff works well(unlike visual D) but it's not stuff 
I can't live without or make programming faster. I might not be 
using it's features though(I'm relatively new to it so). To me it 
just feels much better than Visual D because it works well and 
everything seems to be integrated to work for the programmer and 
the experience. but this is the difference between probably 
thousands of people working on it vs 1.




>>
>> I'm wondering if it would be worth the effort to try to get D 
>> to work with android studio since it can compile to all those. 
>> E.g., write the jetpack compose code in kotlin and hook up the 
>> business in with D somehow. If it wasn't too much trouble and 
>> worked well then this might give one the best of both worlds. 
>> JNI is a bit of a PIA when trying to call java from C because 
>> one has to use reflection for every aspect and it is very long 
>> winded. This probably could be simplified greatly with some D 
>> meta programming.
>
> You're absolutely correct that with D, you could easily 
> generate stuff like JNI bindings.
>
> But notice that JNI is going to be replaced with a new 
> solution: 
> https://docs.oracle.com/en/java/javase/21/core/foreign-function-and-memory-api.html
>
> This is still in preview but you may want to look at it if you 
> find JNI too horrible.

Thanks. I don't find JNI that bad. It's a little bit of a pain to 
deal with "marshaling" but it's surprisingly worked well for what 
I have done even stumbling around. Basically I only ran in a few 
problems later on. I did use jni_bind though which helped a 
little setting up the structures.

I ran into a few problems when trying to call java functions but 
once I worked out precisely what it really wanted it then all 
just worked(of course I have no idea if it's leaking memory or 
whatnot but it's not crashing so ;)).

I basically was just using ffmpeg to decode some audio and was 
able to get it all to work quite easily. Normally, in D, you can 
really struggle with some of these things. It may be because 
there is more info online(but not enough but still more) and it's 
enough to avoid the major pitfalls.

>
> Regarding whether D could be a good option for Android, I very 
> much doubt it. You can't remove the nice tools people are used 
> to in the Java/Kotlin world, if you try you will get a very 
> large backlash... and I'm afraid D will never have as good 
> tools even if it some day gets huge corporate backing because 
> the language was not designed to allow that to be possible 
> (maybe it is possible, but I doubt it's actually feasible - 
> correct if I am wrong), from the list of features it is very 
> clear that was absolutely never a high priority concern.

Anything is possible...well, almost anything. Every language is 
the same.


Because the D metaprogramming is done at compile time and is 
effectively just "generating code" it would have no issues 
integrating in except that any dependencies on that generated 
code would have to obviously come afterwards or two passes would 
have to be used.

It might be a bit of work but it could be done. The cost/reward 
though is significant and unfortunately many people are very 
myopic specially when their $$$ depends on being so.

My logic, and it should be the logic of everyone in theory, is 
that if it only adds to the capabilities then it can't hurt 
anyone(except for, of course, bugs and such that may be 
introduced).

Through the years I have asked for feature requests in various 
languages and apps and the single most common response was "I 
don't need it" as if that is a relevant argument as to why it 
should or shouldn't be implemented.

Most people are not going to find the need for anything until 
they need it and with an attitude like that nothing would ever be 
added. After all, when meta programming was first thought of 
there were plenty of "people"(in quotes because I don't 
technically know if they exist) who said "why would you ever need 
anything like that! Nope, bad idea. It's useless, I don't see any 
need for it!".

It is the way of the world. Progress is made though. Slowly. I 
guess that isn't a bad thing. It's a little frustrating when 
people do claim that something has no use when they can't prove 
it in any way, shape, or form but just their opinion based on 
their ignorance of what is actually being discussed. (after all, 
what is the use of a new born baby?)


The fact of the matter is though that abstractions is how we 
think as humans and it's been a long struggle to get programming 
languages(which are just things that translate math/logic into 
"computer-speak"(bits)) where they are now. I think more people 
are open to the expansive nature than ever before but that 
resistance always shows up in people in different areas. I guess 
it is a "fear of change" type of thing.