Web development status

Mon Jan 22 09:04:48 PST 2007

ok, here is some explanation of the framework I use: Sorry for the length of the post. I promise I will never make one this big again. It took quite some time to write.

1. most of the core is somewhat contract programmed and heavily unittested -- that is, I'm pretty sure it won't break, and the few times I've got a few good ideas, the unittests told me they weren't so good.

2. connecting to the content server can be done one of two ways: 1. over fastcgi (I use lighttpd) or 2. through shttpd embedded. Soon, a third may be added, integrating it directly into the kernel -- similar to the red hat content server, and I'm looking into every possible way to have D load up the various modules in real time. (eg. call a function, and all of the .d files are recompiled and reloaded) This could be done similar to derelict style, or maybe through the mixins idea somewhere else in this newsgroup.

3. it's very fast -- yet has tons of opportunities to be faster (possibly double the speed). Originally, I started with straight output using fastcgi's print functions, and string concats. concatenation is very slow, so I moved to a variadic function to print stuff out (cause templates weren't good enough back then). I now use a template system for design, which parses the template into "panels" then stores the byte code generated for later rendering. This decreased performance by 50% -- but in the future, the D-templates could easily parse it, (if it can ever accept flat files for input) and easily generate machine code to do just that. The beauty of templates though, is templates can change, and they won't be reloaded in memory until I push the button. Also, no downtime either. In every example I had MySQL was ALWAYS the limiting factor. On my laptop, if I cached the data in RAM, I could easily do 6000-14000 requests a second, but using mysql, that number dropped to about 1
000 
requests per second (for a dynamic page). Profiling shows that mysql takes about 80%-90% of the time, so improving the speed of templates will yield little results.

4. it's still pretty immature. I will need help getting it super stable and production worthy. There are no bugs that I know of, but that doesn't mean they don't exist.

5. I use custom links and forms, but they are not necessary. This is because the program I'm writing needs everything to go through an iframe or through ajax, so I don't mess with the layers on top. It gets complicated explaining, so I just won't... the standard <a href="?lala=2">link</a> will work quite fine. It's optional. Because of the nature of what I do, the applications must start having more motion. So, one of the directions I'm moving in, is to make it easier to write inline the stuff to mess with the javascript/dom.

6. you will see them... the qry(...) and the prt(...) are hideous... they shouldn't look after a ^ to insert the data. It should be templated version of writef that everything is made on compile time. Later that can be made... for now, who cares... it's fast enough, and old code dies hard.

---------------

first, it searches for main.pnl in the panels dir and starts from there... here are various template parameters that can be seen as examples... HTML is designed to be completely intermingled

<%interface name: "myprofile" %>
<%load Base %>
<%load Me %>
<%if uid == 0 %>
	<b>you are not logged in</b>
<%else%>
	<div>your name is <%=Me.fullname%>
	<%load Users:Friends {friends_gid: $Me.friends_gid } %>
	<div>You have <%=Users.total%> friends</div>
	<%loop Users:Friends %>
	// copy and pasted from one of my projects...
             <div class="friend">
                 <div class="izquierda_amigos">
                     <img src="i/60/<%=Users.pic%>" alt="<%=Users.fullname%>" />
                 </div>
                 <div class="derecha_amigos">
                     <h3><a href="#" class="sub_azul"><%=Users.firstname%> <%=Users.lastname%></a></h3>
                 </div>
             </div>
            <%endloop%>
<%endif%>

you can make/set vars in the templates...
<%variable name: "varname" type: "uint" default: "2" %>
<%set name: "varname" value: 3 %>

linking is easy... in any panel, you can define where a sub panel will go:

// ex: main.pnl
<%interface name: "main" %>
html... blah blah blah
<%panel name: "m" default: "home" %>
more html blah blah blah

then in the url, to put the panel "home" inside of the subpanel location "m" you do nothing... however, let's say that you want the panel named "myprofile" in there. That's done with the URL string:

<a href="?m=myprofile">link to my profile</a>

pretty easy.

for those looking for a truly object oriented design, don't look here... lol, it looks like I'm writing C/PHP mix most of the time, cause of my heavy use of assoc arrays and loops for parsing...

The modules are written in D and are quite easy to make as well... There are three types... Loopable, One-shot (no loops), and read only (for feedback to forms) For example, let me show you what it takes to make the above Users:Friends module (it's a loopable type):

static class Users : TemplateStaticImport {
	static:
	char[] firstname;
	char[] lastname;
	char[] fullname;
	char[] pic;
	char[][][] res;

	uint page;
	uint size;

	const char[] name = "Users";

	static this() {
		// all users in the website
		GenericLoop!(name ~ ":Site", generic_init,
			function () {
				reload = true;
				total = toUint(q1v("SELECT COUNT(*) FROM `users`"));
			},
			function int() {
				if(reload) {
					res = qry("SELECT `uid`, `firstname`, `lastname`, `fullname`, `pic` FROM `users`");
					count = res.length;
				}

				return generic_loop;
			});

		// You can make as many of these as you want in this class actually... technically, you don't even need it in a class, but it is for now here in a class, because they used to be instantiated, not static...

		// just the group group of people I ask (which happen to be friends of the person)
		GenericLoop!(name ~ ":Friends", generic_init,
			function () {
				assert(friends_gid);
				assert(*friends_gid);
				reload = true;
				total = toUint(q1v("SELECT COUNT(*) FROM `group_members` WHERE `type` = '" ~ s_gid_user_friends ~ "' AND `gid` = '^' AND `id` != '^'", *friends_gid, uid));
			},
			function int() {
				if(reload) {
					assert(friends_gid);
					assert(*friends_gid);
					// I know it's more optimized to do a direct relations table here, but that's not what we need... we need generic groups of people for queries like this all over the place :) plus... mysql 5.1 partitions will solve my problems almost entirely
					res = qry("SELECT u.`uid`, `firstname`, `lastname`, `fullname`, `pic` FROM `group_members` LEFT JOIN `users` u ON (u.`uid` = `id`) WHERE `type` = '" ~ s_gid_user_friends ~ "' AND `gid` = '^' AND `id` != '^'", *friends_gid, uid);
					count = res.length;
				}

				return generic_loop;
			});
	}

	uint* friends_gid;
	uint this_init(PNL* pnl, void*[char[]] args) {
		pnl.registerString(name ~ ".firstname", &firstname);
		pnl.registerString(name ~ ".lastname", &lastname);
		pnl.registerString(name ~ ".fullname", &fullname);
		pnl.registerString(name ~ ".pic", &pic);

		if("friends_gid" in args) {
			friends_gid = cast(uint*)args["friends_gid"];
		}

		return 0;
	}

	void this_set() {
		Base.zid = toUint(res[current][0]);
		firstname = res[current][1];
		lastname = res[current][2];
		fullname = res[current][3];
		pic = res[current][4];
	}

	mixin Generic!(this_init, this_set) G;
	alias G.loop generic_loop;
	alias G.pnl_init generic_init;
	alias G.current current;
	alias G.reload reload;
	alias G.total total;
}

The other thing that's written in D are the functions (things that modify the database). Our database doesn't use cluster, it uses replication, so every write has to go through the single master -- for this reason, all qry (full table), q1r (one row), q1c (one column) and q1v (one value) talk to the database at localhost, and q(for writing) talks to the master. q(...) can only be found in functions.d ... which is good to prevent possible failures.

functions are simple. All data coming in comes to the POST/GET associative array, and the function is called with the query or post string func=logout or func=postwcomment

case "logout":
	if(uid) {
		q("UPDATE `session` SET `online` = 0 WHERE `uid` = ^ AND `sid` = ^", uid, sid);
		q("UPDATE `users` SET `last_logout` = ^ WHERE `uid` = ^", request_time, uid);
		uid = zid = 0;
		is_me = false;
	}
break;
case "postwcomment":
	if(uid) {
		char[]* wcomment = ("wcomment" in POST);
		if(wcomment) {
			// There's no injection here, strings are automatically escaped with the query functions... :)
			q("INSERT INTO `profilewall` ( `profile_uid` , `poster_uid` , `wid` , `text`, `time` )"
					"VALUES ('^', '^', '', '^', '^');", zid, uid, *wcomment, request_time);
		}
	}
break;

Still interested? Feedback? Thoughts?