Idea to make D great again!

[Laeeth Isharc]

On Monday, 7 October 2019 at 00:56:51 UTC, PiSquared wrote:
> D has the ability to embed most languages in to it through 
> scripting. There is matplotlibD already, I have modified the 
> code to be able to run arbitrary scripts and also work with 
> Lua. Any language can be used through the method of:
>
> 1. Writing a D string representing the target language's 
> program to a file or stdin of the interpreter.
>
> 2. Before executing above, first modify the input to replace 
> special tokens with D data. This allows one to insert D data in 
> to the target program. This allows one to use D to generate the 
> data which may be much faster and can be used in other ways.
>
> [See below for an example]
>
> These methods should work with all languages since it simply 
> emulates coding manually. It becomes a powerful feature because 
> one and leverage the target language as if one was writing 
> directly in the languages.
>
> More importantly, Visual D and Visual Studio allow one to load 
> many target languages in to a D project and get almost full 
> benefit such as intelligence(debugging does not work but I 
> imagine it could with some work).
>
>
> Furthermore, with some work, interopt can be made to work 
> really well by transferring data between the programs using a 
> client server protocol which can be done through files or 
> global memory(depending on the languages level of OS access).
>
>
> With the proper design 1. D can be used along side and as a 
> host for most languages. The most common will work fine. With a 
> bit of work it should be work extremely well. This allows D to 
> be leverage and piggy back on these languages and will bring 
> users of those languages to D. D can be used for what it is 
> good at and the holes it has can be filled by some other 
> language. 2. The design is not specific to D as most languages 
> have this ability, but with D it generally can be quite easy.
>
> The only downside is that of "marshaling" the data, this can be 
> reduced significantly by not using text to transfer data but by 
> having appropriate data marshalers that can work in memory and 
> relate data between D and the target language correctly.
>
>
> Further more, the target language needs some capacity too send 
> data back to the D program.
>
> I have created a server and a client that do this. It is used 
> for writing latex code in D. This allows me to also use 
> python's scientific plotting libraries along with tikz by using 
> a common D library. Tikz is slow, D+mlab is fast.
>
> The latex code, for example, can send data back to the server 
> using a client proxy and this can trigger different results 
> when the latex file is being compiled by latex.
>
> It is all quite simple but it allows me to debug D programs in 
> D without issue as all the plumbing on the target language is 
> relatively hidden.
>
>
> I would like you guys to consider taking this proposal 
> seriously: Create a proper D design that allows one to leverage 
> any target language in a unified way. Make it robust, powerful, 
> feature rich.
>
> It will bring many people to use D if done properly. With a 
> proper IDE D could act as a centerpiece to use many languages 
> together.
>
> One could use a gui in lua, a plotting library in python, and 
> latex dox generation.
>
> All done rather seamless.
>
>
>
> here is a lua target, bare bones with no fancy features and 
> essentially ripped from matplotlibD(which essentially does the 
> same with python, of which I have a version that is essentially 
> the same as below):
>
>
> module LuaD;
> import std.stdio;
>
> enum DDataToken = `"@D$`;
>
> alias immutable bool LuaBool;
> alias immutable (void*) LuaNone;
>
> LuaBool False = false;
> LuaBool True = true;
> LuaNone None = null;
>
> // A header to use for common lua files
> string luaHeader = `
> `;
>
> string luaFooter = ``;
>
>
> string d2lua(T)(T v)
> {
>
> 	int x = 3242;
>     import std.conv, std.traits, std.array, std.algorithm, 
> std.format: format;
> 	static if (is(T : LuaNone))
> 		return "None";
>
>     else static if (is(T : bool))
>         return v ? "True" : "False";
>
>     else static if (is(T : string))
>         return format("\"%s\"", v);
>
>     else
> 	{
> 		static if (isArray!T)
> 			return v.map!(a=>to!string(a)~",").join;
>
>         return format("%s", v);
> 	}
> }
>
>
>
> string lua_path = `C:\Lua\lua.exe`;
>
> // executes lua on the program
> void lua(string program)
> {	
>
>     import std.process: environment, pipeProcess, wait, 
> Redirect;
>
> 	if (!lua_path)
> 		lua_path =  environment.get("LuaD", "lua");
> 	
>     auto pipes = pipeProcess(lua_path, Redirect.stdin | 
> Redirect.stderr);
>
>     pipes.stdin.writeln(program);
>     pipes.stdin.writeln("exit()");
>     pipes.stdin.close();
>
>     auto result = wait(pipes.pid);
>
>     if (result != 0) {
>         string error;
>         foreach (line; pipes.stderr.byLine)
>             error ~= line ~ "\n";
>         throw new Exception("\n\nERROR occurred in Lua:\n" ~ 
> error);
>     }
> }
>
>
>
> // converts all "@$..." strings to D variables by converting
> auto luaD(bool ret = false)(string program)
> {	
>
> 	if (program.length < 5) return "";
>
> 	import std.string, std.range, std.algorithm;
>
>
> 	string r;
> 	for(int i = 0; i < program.length; i++)
> 	{
> 		if (i < program.length - 5)
> 		{
> 			if (program[i..i+DDataToken.length] == DDataToken)
> 			{
> 				i += DDataToken.length;
> 				string id;
> 				while(i < program.length && program[i] != '"')
> 					id ~= program[i++];
>
> 				r ~= `"~d2lua(`~id~`)~"`;
> 				continue;
> 			}
> 		}
> 		if (program[i] == '"') r ~= "\\";
> 		r ~= program[i];
> 	}
>
> 	static if (ret) { return `"`~r~`"`; }
>
> 	return `lua("`~r~`");`;
> }
>
>
>
> Then a simple lua program `test.lua`:
>
>     print("@D$TestString")
>
>
>
> one calls this program in D:
>
>     auto TestString = "This is a D string";
>
>     // Will automatically substitute TestString
>     mixin(luaD(import(test.lua));
>
>
>
>
>
> and the lua program will be executed with the D data.
>
>
> I have used such things for computing data in D and then 
> plotting them using some of pythons plotting 
> libraries(matplotlib, mlab, mayavi, etc).
>
>
> It is not as optimal since large datasets can cause problems 
> since they can take up so much textual space but this can be 
> alleviated by writing them to binary files, or ideally, using 
> memory transfers.
>
>
> With a good robust design D can be the centerpiece of language 
> hosting bringing a very large number of users. Python, for some 
> odd reason, has a lot of scientific plotting libraries but, as 
> most people know, is not very fast. Imagine using D for the 
> algorithms and python for the plotting. That is precisely what 
> I have done and it works well(debugging errors is the hardest 
> since no IDE but that could change). In visual D I get full 
> syntax coloring, intellisense, and documentation for python and 
> lua.
>
> Imagine F#, C#, vbs, C++, Haskell, even assembly and many other 
> languages being usable with this method. With a good 
> "marshaling" library it one would could have limited 
> performance degradation. For apps that do not have performance 
> issues, say typical gui's one can use a language with good gui 
> frameworks and D for the business end. With D's meta 
> capabilities one could probably export most of the bridging 
> automatically in to the gui.
>
>
> Done well it could be a boon to D and bring in many new people 
> and $$$. As far as I'm aware, no language or framework exists 
> to do this well and easily yet it is not a difficult problem.

Yes, and for example in C# and F# I think Roslyn could take you a 
long way since you can compile dotnet code at D runtime.

As is well known, C++ is an interpreted language where you can 
instantiate templates at runtime.

I'm joking, but it can be if you want it to be and CERN do this 
at considerable scale with Cling, which allows you to instantiate 
templates at runtime and introspect, create instance of classes 
and call them.  Some interesting early results in calling cling 
from D, but I was waiting till things were a bit more polished to 
talk about them.  The next stage might be to run cling on clang 
to help with dpp development.  Libclang looks amazing at first 
sight but it is missing a lot.

I think there are two reasons why interoperability matters.  One 
is external libraries but another is that when you are exploring 
using D then it's often a bit challenging if you have C++ code 
and need to use it or implement part of a new feature in D.  If 
replacing working code you don't ideally want to do a complete 
rewrite but replace small pieces at a time and have working code 
all along the way.  That might not be easy in many cases.

I think RoI is often a less important barrier than limits to 
upfront pain.  If you use D you are already an unusual sort of 
person because the discomfort is front loaded and how do you know 
it will be worth it to persevere?  Atila downloaded the compiler 
and back then it segfaulted when he tried to compile hello world! 
  Most reasonable people would have given up then.

But the dollar return on being reasonable is time varying and for 
some people at some times it's not necessarily a bad thing to 
hire people who are a bit unreasonable.  Perhaps all progress 
depends on people not being reasonable.

So if it becomes easier to use libraries and internal code 
written in other languages then it's easy to make small 
beginnings.  Lots of large projects begin as little beginnings 
and so making it easy to start work has value for the ecosystem I 
think.

I agree that sometimes you don't need to do things that are that 
clever to be quite productive.  But one probably can do a bit 
better than what you suggest even if for any individual it's more 
than good enough.

It's almost easy to use python, C, Lua,R libraries from D but not 
quite.  Pyd is showing its age right now and LuaD isn't that well 
maintained.  It's not super easy to find embedr and it's not 
trivially easy to get up and running.

I think lots of that is documentation, maintenance,small 
improvements and blog posts.

Dpp is pretty solid a lot of the time.  On Linux anyway but on 
Windows not yet, judging by a conversation I had yesterday.  I 
asked Adam Ruppe to look into that and I think in time it will 
just work for what it works for.  It's a shame it doesn't 
dependably yet at least handle Linux headers without choking.  I 
think that's an SAoC project if I recall.  We could do a bit more 
to programmatically just try wrapping every header we can get our 
hands on and then try to programmatically analyse the results.

Ultimately there is a limit with macros, maybe with using 
libclang and maybe how far you can get with generating strings in 
D rather than AST to AST transformation.  The latter for C isn't 
a trivial project - DoD spent millions on C2Rust though on the 
other hand they are not going to be able to do things on the 
cheap.  C to D ought to be quite doable though.  Our focus has 
been more on the cpp side for now but in time could do a bit more 
with C.

With PyD under autowrap there is a project called PyNIH.  Steady 
progress with focus first on Python calling D but that helps with 
the reverse.

Embedding C# in D has stalled a bit - see dflat and a project to 
do so via Mono also.  It's not difficult work technically but 
just a bit of a grind.

We did some very basic work on embedding Julia in D.  That also 
isn't difficult but time is limited.

The basic idea is to use Symmetry Integration Language as a 
substrate where you can write plugins in any sensible language 
and call it from any sensible language for us.  We have to solve 
our own problems first but in the process of doing so probably it 
will help others.

One more thing - work Sebastian Koppe did shows a proof of 
concept of D compiling to web assembly and using Javascript 
libraries.  He will generate the wrapping code from Typescript 
bindings in time.

 From work I did before I don't think embedding Java would be so 
difficult but we just don't use Java that much.

So I think the biggest constraint right now is the ability to 
organise and direct the work in some cases (like C#) and in 
others for other people to submit improvements.  But in time I 
think interoperability will just keep getting better.