Ideas for a brand new widget toolkit

[Joakim]

On Wednesday, 14 August 2013 at 02:23:07 UTC, Adam D. Ruppe wrote:
> On Tuesday, 13 August 2013 at 20:33:48 UTC, Joakim wrote:
>> You mentioned X11 to me before, when we talked about this idea 
>> over email.
>
> Ah yes. I think X's biggest problem though is that it doesn't 
> do *enough*. The protocol is fairly efficient for what it does, 
> but it just doesn't do enough so the commands aren't 
> particularly compact when you start to get fancier.
As you point out later, I don't think it's the compactness of the 
commands that is the problem as much as that X11 bites off too 
much.  If you're going to be both efficient and allow remote GUI 
forwarding, that's an incredibly difficult and orthogonal set of 
features to deliver on, which is why Wayland only focuses on 
efficiency.  I'd punt on much of the stuff X11 tries to do with 
my lean GUI runtime, which I'll call Crack from here on out.

> For instance, in my crappygui.d, I used XDrawString to draw 
> text. This was the result: http://arsdnet.net/gui.png
--snip--
> Note btw though, you *can* save the image on the display server 
> to reuse it later, so every time you print a string it doesn't 
> necessarily send that data across, but I'm not sure if the 
> higher level libraries actually do this... the downsides of 
> abstractions is you can miss important details like this.
Interesting, it does highlight what a mess X11 is.  I see no 
reason to build on such an old protocol.

> This is why my simpledisplay.d now has a class Sprite in 
> addition to class Image, and why Qt has QImages and QPixmaps, 
> but how often do you think of drawing text as being something 
> that needs a Pixmap to be efficient? Again the library might 
> handle it, I'm not sure, but the relatively pathetic 
> performance of a lot of apps on a remote X connection makes me 
> think they probably don't.
This is why I'd implement text layout and rendering in the Crack 
client, with the server simply passing along strings for the 
client to render, breaking the simple OpenGL wrapper promised 
before. ;) Here's some other exceptions I'd put in, which I 
mentioned to you before:

Text input - As you say below, you wouldn't want to hit the 
server for every keypress, so text input boxes would be handled 
by the client.  It would be easy to implement since you're 
already doing text layout and rendering in the client.

Scrolling - Just as with text input, if you're using one 
rectangle onscreen as a scrollbar, you usually wouldn't want to 
hit the server just to get the new scroll position of the other 
rectangle it controls.  There would be some scrolling-related 
event-handling in the Crack client, so that you could designate 
one rectangle as the scrollbar control for another rectangle and 
the client would handle scrolling, simply notifying the server of 
the new scroll position after the user is done scrolling.

Upload/Download progress -  Another ability you'll need is to 
update a rectangle based on the progress of files being 
downloaded or uploaded, so you can create a progress bar without 
having to constantly hit the server to get the status of file 
transfers.

File chooser - And of course, access to the native file chooser 
widget.

Caching - Finally, you'll want some resource and event-handling 
caching so that you can load updates quicker, just like AJAX once 
hacked into the web stack.  Say you're implementing a photo 
gallery, you'll have the Crack client download the next photo in 
the gallery and be able to tell the client that if a certain 
rectangle is clicked, it should swap in the next photo without 
checking with the server first.  This is similar to your 
"predictive paths" idea from DWS and would greatly decrease 
network traffic and GUI lag.

The caching of event handling would be narrowly circumscribed, 
boolean logic with mouse/keyboard events as input and GUI updates 
as output, so that it's simpler and much more secure than a full 
language interpreter like javascript.

Obviously, all these exceptions break the simple OpenGL wrapper 
that I'd wanted Crack to be, but the goal is to minimize such 
exceptions, so that you don't become the bloated beast that the 
web client has become today. :)

> Anyway, the other thing too is all events go to the client 
> application from the display server, and then the application's 
> changes go back to the display server. Since X itself doesn't 
> offer any kind of widgets, a text control for instance would 
> work like this:
--snip--
> So I'd want to do higher level events too, and the application 
> can request them. For instance, if all you want is a basic text 
> input, output something like <textarea></textarea> and let the 
> display do the details.
I agree, text input boxes should be handled by the Crack client.

> (Another really nice benefit here, if we do it right, is it 
> could use native controls on systems like Windows, or even pipe 
> it to an external app on unix, and get a nice customized, 
> adaptable system. Though while it sounds simpler in ways, this 
> is easier said than done.)
I was thinking along similar lines for other widgets, like a 
video player.  I don't think video should be in the initial 
release, too complex, but just as HTML5 eventually added video 
functionality, you're going to want it in Crack someday.

The way I formulated it is that users would choose from various 
video player widgets and that widget would come up in any app 
that played video, ie the video player is out of the control of 
the app developer.  This avoids the problem you see with video on 
the web today, where you are presented with dozens of different 
video player implementations, whether implemented in Flash, 
Silverlight, or HTML5, all broken in their own special ways. ;)

The app simply sends a video file to the Crack client and then 
the video player widget of the user's choice kicks in.  This 
video player widget would be written in D and the user would be 
given a choice of which player widget best suits them when they 
download the Crack runtime.  Alternatively, you could use 
plugins, but the key is to tightly control where these player 
widgets could come from, so that you aren't running untrusted 
code in the runtime.

Perhaps the same choice could be offered with text input boxes 
and other functionality in the client.

> With these higher level events and widgets, unless you need to 
> override some event, it can just be handled without the round 
> trip and improve response time on slow connections.
Yeah, you do need local logic or widgets for some special cases, 
no doubt about it.

> But, indeed, we don't want to go *too* far either, especially 
> since then we'd end up with a web browser situation where 
> people write their applications in the scripting language...
Agreed, you need to keep tight limitations, or things get out of 
hand into the mess that the web stack is today.  This is why I 
wouldn't put javascript or any other scripting language in Crack.

>> What basic widgets do you have in mind, to keep on the 
>> client-side?  Also, just widgets in the client or some basic 
>> layout too?
>
> Layout would be nice too. Ideally, I'd love if my apps worked 
> on both guis and text mode uis and laying them out would be a 
> bit different.
I want to keep as much layout as possible out of the Crack 
client, and I don't think this warrants it.

> For my crappygui.d, I'm aiming to do:
>
> menus, labels, radio box, checkbox, buttons, grid layouts, text 
> input, slider, number chooser, list boxes, and basic 2d 
> drawing. Pretty much the basic stuff you get on html forms. 
> Maybe more later, but I want to actually be able to get this 
> working over the next weekend or two, so I'm keeping it simple. 
> (And right now I'm not actually doing network transparency, 
> I'll come back to that, my basic idea there is to simply 
> forward all the function calls with rpc.)
Most, but not all, of those can be done with just rectangles 
directed from the server in Crack, with the aforementioned 
exceptions of sliders and text input.  As for basic 2D drawing, I 
see no worthwhile use case.  We've had the canvas tag in the 
browser for years now, anyone really using it?

>> it's more complicated to code all that reattaching 
>> functionality and isn't necessary for most apps, most app devs 
>> don't bother.
>
> If we do it right, it will be zero effort :) GNU Screen has 
> pretty much pulled it off for unix terminal programs. GUIs have 
> Remote Desktop and friends... rdp is amazing btw, Microsoft (or 
> whoever wrote it originally and sold it to them) did an 
> excellent job and is a decent counter argument to me -  I think 
> remote desktop is a simple viewer, the rdesktop app on unix 
> isn't a particularly large piece of code and works quite well, 
> but still I like my way.
While remote desktop is decent, it's trying to do too much: 
mirroring an entire desktop is overkill.  Better to use a lean 
client that handles most situations.