BNF grammar for D?

[Kingsley via Digitalmars-d]

On Sunday, 21 December 2014 at 00:34:06 UTC, Kingsley wrote:
> On Friday, 19 December 2014 at 02:53:02 UTC, Rikki Cattermole 
> wrote:
>> On 19/12/2014 10:19 a.m., Kingsley wrote:
>>> On Wednesday, 17 December 2014 at 21:05:05 UTC, Kingsley 
>>> wrote:
>>>>
>>>>> Hi Bruno,
>>>>>
>>>>> Thanks very much. I do have a couple of questions about DDT 
>>>>> in
>>>>> relation to my plugin.
>>>>>
>>>>> Firstly - I'm not too familiar with parsing/lexing but at 
>>>>> the moment
>>>>> the Psi Structure I have implemented that comes from the DDT
>>>>> parser/lexer is not in any kind of hierarchy. All the 
>>>>> PsiElements are
>>>>> available but all at the same level. Is this how the DDT 
>>>>> parser
>>>>> works? Or is it down to my implementation of the 
>>>>> Parser/Lexer that
>>>>> wraps it to create some hierarchy.
>>>>>
>>>>> For intellij it's going to be vastly easier to have a 
>>>>> hierarchy with
>>>>> nested elements in order to get hold of a structure 
>>>>> representing a
>>>>> class or a function for example - in order to do things 
>>>>> like get the
>>>>> start and end lines of a class definition in order to apply 
>>>>> code
>>>>> folding and to use for searching for classes and stuff.
>>>>>
>>>>> Secondly - how active it the development of DDT - does it 
>>>>> keep up
>>>>> with the D2 releases.
>>>>>
>>>>> --Kingsley
>>>>
>>>> After doing a bit more research it looks like I have to 
>>>> create the psi
>>>> hierarchy myself - my current psi structure is flat because 
>>>> I'm just
>>>> converting the DeeTokens into PsiElements directly. I've 
>>>> still got
>>>> some experimentation to do. On the plus side I implemented 
>>>> commenting,
>>>> code folding but everything else needs a psi hierarchy
>>>
>>> I've done some more investigation and I do need to build the 
>>> parser
>>> myself in order to create the various constructs. I've made a 
>>> start but
>>> I haven't gotten very far yet because I don't fully 
>>> understand the
>>> correct way to proceed.
>>>
>>> I also had a look at using the DeeParser - because it already 
>>> does most
>>> of what I want. However the intellij plugin wants a PsiParser 
>>> which
>>> returns an intellij ASTNode in the primary parse method. I 
>>> can't see an
>>> easy way to hook this up with DeeParser because the 
>>> ParsedResult
>>> although had a node method on it - gives back the wrong type 
>>> of ASTNode.
>>>
>>> Any pointers on how I might get the DeeParser to interface to 
>>> an
>>> intellij ASTNode would be appreciated.
>>
>> Read my codebase again, it'll answer a lot of questions. Your 
>> parser is different, but what it produces shouldn't be. and 
>> yes it supports hierarchies.
>
> Hi
>
> So finally after a lot of wrestling with the internals of 
> intellij I finally managed to get a working parser 
> implementation that produces a psi hierarchy based on the 
> DeeParser from the ddt code.
>
> The main issue was that Intellij only wants you to create a 
> parser using their toolset - which is either with a BNF grammar 
> that you can then generate the parser - or with a hand written 
> parser. Since I'm already using the DDT lexer and there is a 
> perfectly good DDT parser as well - I just wanted to re-use the 
> DDT parser.

Hi Bruno - would be easy to return the list of tokens included 
for each node in the DeeParser?
>
> However Intellij does not provide any way to create a custom 
> AST/PSI structure or use an external parser. So I basically had 
> to wrap the DeeParse inside the Intellij parser and sync them 
> up programmatically. It's not the most efficient way in the 
> world but it at least works.
>
> In the long term I will write a BNF grammar for Intellij (using 
> their toolkit) but I can see that will take me several months 
> so this is a quick way to get the plugin up and running with 
> all the power of intellij extras without spending several 
> months stuck learning all about the complexities of grammar 
> parsing and lexing.
>
> Thanks very much for you help. Once I get a bit more of the 
> cool stuff done I will release the plugin.