BNF grammar for D?

[Kingsley via Digitalmars-d]

On Friday, 19 December 2014 at 02:53:02 UTC, Rikki Cattermole 
wrote:
> On 19/12/2014 10:19 a.m., Kingsley wrote:
>> On Wednesday, 17 December 2014 at 21:05:05 UTC, Kingsley wrote:
>>>
>>>> Hi Bruno,
>>>>
>>>> Thanks very much. I do have a couple of questions about DDT 
>>>> in
>>>> relation to my plugin.
>>>>
>>>> Firstly - I'm not too familiar with parsing/lexing but at 
>>>> the moment
>>>> the Psi Structure I have implemented that comes from the DDT
>>>> parser/lexer is not in any kind of hierarchy. All the 
>>>> PsiElements are
>>>> available but all at the same level. Is this how the DDT 
>>>> parser
>>>> works? Or is it down to my implementation of the 
>>>> Parser/Lexer that
>>>> wraps it to create some hierarchy.
>>>>
>>>> For intellij it's going to be vastly easier to have a 
>>>> hierarchy with
>>>> nested elements in order to get hold of a structure 
>>>> representing a
>>>> class or a function for example - in order to do things like 
>>>> get the
>>>> start and end lines of a class definition in order to apply 
>>>> code
>>>> folding and to use for searching for classes and stuff.
>>>>
>>>> Secondly - how active it the development of DDT - does it 
>>>> keep up
>>>> with the D2 releases.
>>>>
>>>> --Kingsley
>>>
>>> After doing a bit more research it looks like I have to 
>>> create the psi
>>> hierarchy myself - my current psi structure is flat because 
>>> I'm just
>>> converting the DeeTokens into PsiElements directly. I've 
>>> still got
>>> some experimentation to do. On the plus side I implemented 
>>> commenting,
>>> code folding but everything else needs a psi hierarchy
>>
>> I've done some more investigation and I do need to build the 
>> parser
>> myself in order to create the various constructs. I've made a 
>> start but
>> I haven't gotten very far yet because I don't fully understand 
>> the
>> correct way to proceed.
>>
>> I also had a look at using the DeeParser - because it already 
>> does most
>> of what I want. However the intellij plugin wants a PsiParser 
>> which
>> returns an intellij ASTNode in the primary parse method. I 
>> can't see an
>> easy way to hook this up with DeeParser because the 
>> ParsedResult
>> although had a node method on it - gives back the wrong type 
>> of ASTNode.
>>
>> Any pointers on how I might get the DeeParser to interface to 
>> an
>> intellij ASTNode would be appreciated.
>
> Read my codebase again, it'll answer a lot of questions. Your 
> parser is different, but what it produces shouldn't be. and yes 
> it supports hierarchies.

Hi

So finally after a lot of wrestling with the internals of 
intellij I finally managed to get a working parser implementation 
that produces a psi hierarchy based on the DeeParser from the ddt 
code.

The main issue was that Intellij only wants you to create a 
parser using their toolset - which is either with a BNF grammar 
that you can then generate the parser - or with a hand written 
parser. Since I'm already using the DDT lexer and there is a 
perfectly good DDT parser as well - I just wanted to re-use the 
DDT parser.

However Intellij does not provide any way to create a custom 
AST/PSI structure or use an external parser. So I basically had 
to wrap the DeeParse inside the Intellij parser and sync them up 
programmatically. It's not the most efficient way in the world 
but it at least works.

In the long term I will write a BNF grammar for Intellij (using 
their toolkit) but I can see that will take me several months so 
this is a quick way to get the plugin up and running with all the 
power of intellij extras without spending several months stuck 
learning all about the complexities of grammar parsing and lexing.

Thanks very much for you help. Once I get a bit more of the cool 
stuff done I will release the plugin.