Idea to make D great again!

[PiSquared]

D has the ability to embed most languages in to it through 
scripting. There is matplotlibD already, I have modified the code 
to be able to run arbitrary scripts and also work with Lua. Any 
language can be used through the method of:

1. Writing a D string representing the target language's program 
to a file or stdin of the interpreter.

2. Before executing above, first modify the input to replace 
special tokens with D data. This allows one to insert D data in 
to the target program. This allows one to use D to generate the 
data which may be much faster and can be used in other ways.

[See below for an example]

These methods should work with all languages since it simply 
emulates coding manually. It becomes a powerful feature because 
one and leverage the target language as if one was writing 
directly in the languages.

More importantly, Visual D and Visual Studio allow one to load 
many target languages in to a D project and get almost full 
benefit such as intelligence(debugging does not work but I 
imagine it could with some work).


Furthermore, with some work, interopt can be made to work really 
well by transferring data between the programs using a client 
server protocol which can be done through files or global 
memory(depending on the languages level of OS access).


With the proper design 1. D can be used along side and as a host 
for most languages. The most common will work fine. With a bit of 
work it should be work extremely well. This allows D to be 
leverage and piggy back on these languages and will bring users 
of those languages to D. D can be used for what it is good at and 
the holes it has can be filled by some other language. 2. The 
design is not specific to D as most languages have this ability, 
but with D it generally can be quite easy.

The only downside is that of "marshaling" the data, this can be 
reduced significantly by not using text to transfer data but by 
having appropriate data marshalers that can work in memory and 
relate data between D and the target language correctly.


Further more, the target language needs some capacity too send 
data back to the D program.

I have created a server and a client that do this. It is used for 
writing latex code in D. This allows me to also use python's 
scientific plotting libraries along with tikz by using a common D 
library. Tikz is slow, D+mlab is fast.

The latex code, for example, can send data back to the server 
using a client proxy and this can trigger different results when 
the latex file is being compiled by latex.

It is all quite simple but it allows me to debug D programs in D 
without issue as all the plumbing on the target language is 
relatively hidden.


I would like you guys to consider taking this proposal seriously: 
Create a proper D design that allows one to leverage any target 
language in a unified way. Make it robust, powerful, feature rich.

It will bring many people to use D if done properly. With a 
proper IDE D could act as a centerpiece to use many languages 
together.

One could use a gui in lua, a plotting library in python, and 
latex dox generation.

All done rather seamless.



here is a lua target, bare bones with no fancy features and 
essentially ripped from matplotlibD(which essentially does the 
same with python, of which I have a version that is essentially 
the same as below):


module LuaD;
import std.stdio;

enum DDataToken = `"@D$`;

alias immutable bool LuaBool;
alias immutable (void*) LuaNone;

LuaBool False = false;
LuaBool True = true;
LuaNone None = null;

// A header to use for common lua files
string luaHeader = `
`;

string luaFooter = ``;


string d2lua(T)(T v)
{

	int x = 3242;
     import std.conv, std.traits, std.array, std.algorithm, 
std.format: format;
	static if (is(T : LuaNone))
		return "None";

     else static if (is(T : bool))
         return v ? "True" : "False";

     else static if (is(T : string))
         return format("\"%s\"", v);

     else
	{
		static if (isArray!T)
			return v.map!(a=>to!string(a)~",").join;

         return format("%s", v);
	}
}



string lua_path = `C:\Lua\lua.exe`;

// executes lua on the program
void lua(string program)
{	

     import std.process: environment, pipeProcess, wait, Redirect;

	if (!lua_path)
		lua_path =  environment.get("LuaD", "lua");
	
     auto pipes = pipeProcess(lua_path, Redirect.stdin | 
Redirect.stderr);

     pipes.stdin.writeln(program);
     pipes.stdin.writeln("exit()");
     pipes.stdin.close();

     auto result = wait(pipes.pid);

     if (result != 0) {
         string error;
         foreach (line; pipes.stderr.byLine)
             error ~= line ~ "\n";
         throw new Exception("\n\nERROR occurred in Lua:\n" ~ 
error);
     }
}



// converts all "@$..." strings to D variables by converting
auto luaD(bool ret = false)(string program)
{	

	if (program.length < 5) return "";

	import std.string, std.range, std.algorithm;


	string r;
	for(int i = 0; i < program.length; i++)
	{
		if (i < program.length - 5)
		{
			if (program[i..i+DDataToken.length] == DDataToken)
			{
				i += DDataToken.length;
				string id;
				while(i < program.length && program[i] != '"')
					id ~= program[i++];

				r ~= `"~d2lua(`~id~`)~"`;
				continue;
			}
		}
		if (program[i] == '"') r ~= "\\";
		r ~= program[i];
	}

	static if (ret) { return `"`~r~`"`; }

	return `lua("`~r~`");`;
}



Then a simple lua program `test.lua`:

     print("@D$TestString")



one calls this program in D:

     auto TestString = "This is a D string";

     // Will automatically substitute TestString
     mixin(luaD(import(test.lua));





and the lua program will be executed with the D data.


I have used such things for computing data in D and then plotting 
them using some of pythons plotting libraries(matplotlib, mlab, 
mayavi, etc).


It is not as optimal since large datasets can cause problems 
since they can take up so much textual space but this can be 
alleviated by writing them to binary files, or ideally, using 
memory transfers.


With a good robust design D can be the centerpiece of language 
hosting bringing a very large number of users. Python, for some 
odd reason, has a lot of scientific plotting libraries but, as 
most people know, is not very fast. Imagine using D for the 
algorithms and python for the plotting. That is precisely what I 
have done and it works well(debugging errors is the hardest since 
no IDE but that could change). In visual D I get full syntax 
coloring, intellisense, and documentation for python and lua.

Imagine F#, C#, vbs, C++, Haskell, even assembly and many other 
languages being usable with this method. With a good "marshaling" 
library it one would could have limited performance degradation. 
For apps that do not have performance issues, say typical gui's 
one can use a language with good gui frameworks and D for the 
business end. With D's meta capabilities one could probably 
export most of the bridging automatically in to the gui.


Done well it could be a boon to D and bring in many new people 
and $$$. As far as I'm aware, no language or framework exists to 
do this well and easily yet it is not a difficult problem.

Also it lets a person become "multi-lingual" in real time. I know 
quite a few languages but rarely use them because I typically use 
only one at a time unless required.  But being able to 
essentially choose the best language for the job would be nice.

Diet templates sort of approach the idea the same but for html. 
The above approach could allow embedding D code in to other 
languages too by having "code strings".

By having a universal design it could be done quite effectively. 
I've already been able to do far more in D by leveraging python 
than I could do otherwise... but by leveraging I mean having D 
automatically transfer in to the python program rather than 
having to do complicated coding.


My code is not robust as it is proof of concept and meets my 
limited needs. I'd like to see a more robust framework for doing 
this... I'd do it myself but I simply do not have the time to do 
it all and make it as good as it should be.