Dynamic Code in D

[CptJack]

Michael Wilson Wrote:
> 
> > For that purpose CommonLisp looks like the a good language (beside
> > Java itself). It's fast enough if used well, and it compiles the
> > functions on the fly. There are some disadvantages, like it's not
> > easy to find CLisp programmers.

He meant "Common Lisp programmers". "CLisp" is just a single 
implementation.

> It's even harder to find D developers, but it's easy for developers
> who know C/C++/C#/Java to learn D (hopefully! since I'm learning it
> and if I used it on a production system other developers at the
> company would have to as well). Lisp, not so much. I guarentee that
> if I said 'I want to reimplement in Lisp' the answer would be no,
> while I received provisional approval for using D on the basis that
> 'it's very similar to Java, our developers can learn it easily'.

That's a valid argument. However, I would like to point you to this
very interesting essay before dismissing Lisp entirely for that reason:
http://www.paulgraham.com/avg.html

> 
> Aside from that the other problems with using Lisp in this app are;
> 1) /Much/ more work to port the sections of code I want to reuse
> from the old system across.

Once you begin to do some serious programming in Lisp, and know it 
as well as you do your current language of choice, I think you will 
find that it is /much/ faster to program in than any other language. 
At first you will be fighting the syntax and the parens, then one day
you will just "get it" and your world will change. It has so many
unique features that you will wonder how you ever programmed without 
them (in fact, almost every "exciting new feature" in language X was implemented in Lisp 40 years ago).


> 2) I don't know of any CL implementations with SSE intrinsics. So
> unless someone else knows one I'd be stuck linking to C code again.
> It's possible that there's a CL implementation with less overhead
> for linking dynamic code to external C libraries than JNI has, but
> if so I don't know of it.

As for SSE support, I know SBCL has SSE support, and therefore I 
believe CMUCL also does (since SBCL is forked from CMUCL). ECL is
completely based around linking dynamic code, and is highly optimized
to do so.

> 3) AFAIK similar issues with trying to do NUMA optimisation as Java
> and worse synchronisation performance (though again, there may be
> some super-optimal CL implementation out there I don't know about).

Again, take a better look at SBCL and CMUCL.

> CptJack wrote:
>  > Everything you need to do is provided by Common Lisp, and I don't mean
>  > "CLisp", which is a byte-code interpreted CL implementation. Contrary
>  > to what most people assume, almost all CL implementations compile
>  > functions on-the-fly to machine code. That's raw machine object code,
>  > not byte-code, compiled and loaded dynamically while the system is
>  > still running. This sounds exactly like what you want.
> 
> That sounds exactly like what the Java version is already doing.

This is my point exactly. You've had to re-implement Lisp to get Java to do what you want, a-la Greenspun's Tenth Rule.
 
> The only obvious benefit with Lisp is a slight simplification of the GP
> engine output stage. As I've noted, Hotspot is very well optimised
> these days, I'm not aware of a CL implementation that does better and
> indeed most perform significantly worse.

http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=sbcl&lang2=java
They look pretty similar to me in this admittedly fairly worthless 
comparison.
 
> D would have the advantage of
> being able to link to C with effectively zero overhead, do manual memory
> allocation and synchronization where beneficial (i.e. NUMA-optimised
> memory allocation that still plays well with the GC) and be able to use
> inline assembly for SSE code. CL doesn't have any of those advantages

Not true. I think it's just a matter of common misconceptions about Lisp and its implementations. Most people think it's slow and interpreted. It's neither. Movitz for example is one Common Lisp implementation designed to be as "on-the-metal" as you want including inline assembly. Some of this movitz code has been ported to other implementations as well.

> plus as I mentioned earlier it's (much) harder to get new
> developers for and (significantly) harder to train existing developers
> in. 

As I said, that is indeed a valid argument. However, some of this is because of perceived hurdles that really don't exist (or are at least smaller than they at first appear). 

The only reason I'm still making arguments in favor of Common Lisp for your project is because what you describe is basically a Common Lisp implementation hacked together from Java and C. Why not just do away with the hacks?