FUTURE LOGIC INC.

2009-07-09

Welcome to my page of Notes, Tips and Tricks for web2py on Google App Engine (aka GAE).

I am your 'host', David Lypka. (email: dlypka@futurelogic.com)

My company is Future Logic Inc, registerd in Ontario, Canada.

This page is running on the free Google App Engine Server.

The information in this page is coming from the 'Big Table' database which is part of GAE.

It is *NOT* coming from a .html file.

Since I have a long professional background as an ASP.NET developer, I hope I can bring a perspective useful for

explaining to the .NET community the how and why of using web2py.

These notes are targeted at running web2py on the Google App Engine.

The 'py' in web2py indicates that it is written in the Python language.

The creator of web2py and T3 Massimo Di Pierro can be seen in this video:

http://us.pycon.org/2009/conference/schedule/event/60/

So, 2 technologies have to be covered and that means covering 2 different forums (at a minimum!):

The Google App Engine forum for Python http://groups.google.com/group/google-appengine-python

and

The web2py developer's forum (English) http://groups.google.com/group/web2py

Of course, the 'maximum' is to also cover any other relevant information out on the Cloud...

I will begin these notes by starting for a reader who knows Microsoft technologies and at least

ASP.NET in particular and yet very little about GAE, Python, or web2py.

I imagine your first big question is: What are the limitations of this web2py-GAE platform?

Answer: As of today April 26, 2009, we are 3 weeks away from a major change in limitations by the GAE Team,

but at this moment, the limitations of application resources on GAE are:

From http://code.google.com/p/googleappengine/issues/detail?id=161&q=reporter:ianbicking&colspec=ID%20Type%20Status%20Priority%20Stars%20Owner%20Summary

The limit is 1000 code files (python .py files) and 1000 static files.

Each file may be up to 10MB.

Also, the total size of all code files must remain under 150 MB.

NOTE for static files, this implies:

From http://www.niallkennedy.com/blog/2009/02/google-app-engine-119.html

Developers can deploy up to 1000 static files of up to 10 MB each, creating up to 10 GB of geo-distributed

static file storage per App Engine instance.

*BEWARE! Note that API requests (e.g. memcache.set(), db.put()) are still limited to 1MB in size

And yes, it is possible (apparently) to buy an IDE which will give you a Visual Studio - like experience with web2py:

http://groups.google.com/group/web2py/browse_thread/thread/144cf755a8c88c0e#

web2py performs automatic database migration when it detects a database schema change, subject to some limitations which are discussed here:

http://groups.google.com/group/web2py/browse_thread/thread/0ff84bd8f9c07b4f

sqlite does not understand migrations well. In particular it
cannot alter table and change a column type. It just stores new values
according to the new type.

On GAE there is no concept of alter table so you can add columns and
you can drop (they will not be dropped, just ignored). You can change
some attributes of a column (like length) since they are ignored
anyway. Some migrations will not work

The sum total of what I have found about web2py forms so far is more or less in these links

http://www.web2py.com/AlterEgo/default/show/205 Custom forms with SQLFORM

http://www.web2py.com/AlterEgo/default/show/92 How to make a good page layout

http://www.web2py.com/AlterEgo/default/show/128 How to Build a Form in the Controller Code

http://www.web2py.com/AlterEgo/default/show/146 using form_factory

http://www.web2py.com/AlterEgo/default/show/139 FORM Select options

http://www.web2py.com/AlterEgo/default/show/100 Sharing the same controller in multiple views

http://www.web2py.com/AlterEgo/default/show/11 Extend and include other views ala inheritance

http://www.web2py.com/AlterEgo/default/show/85 Prepopulate a form from specific record

http://www.web2py.com/AlterEgo/default/show/170 Custom form widgets

http://groups.google.com/group/web2py/browse_thread/thread/05c0b01480366044 Override SQLFORM behavior

http://groups.google.com/group/web2py/browse_thread/thread/1944e1c49fc20819 Complex FORM issue

http://groups.google.com/group/web2py/browse_thread/thread/0dfc9d36f63ee242 SQLFORM.accepts() method

http://groups.google.com/group/web2py/browse_thread/thread/78af035f206f30d5 CRUD class questions

The general Controller pattern for forms implementation is:

def my_controller_function:
         # ... stuff....
         form=crud() # now you can stop and inspect all the elements of this
         send_to_view = dict( form )
         return send_to_view

I have learned that you can custom define the CRUD Form by creating the custom form in the Controller code by calls to various HTML helper classes. These classes are defined the web2py source in gluon/html.py.

I am working on a way to embed the HTML Helper calls within the View markup, in a style similar to using <ASP:somecontrol> declarations in ASP.NET web pages. That would make it easier to view the controls within a WYSIWYG designer in the future. The HTML helper classes are roughly what are 'controls' in ASP.NET.

Interesting web2py Code Snippets

Date: Sat, 2 May 2009 14:50:52 -0700 (PDT)
<iframe src="{ {=URL(r=request,f='edit_comment',args=comment.id))} }"></iframe>

Date: Sat, 2 May 2009 16:11:34 -0700 (PDT)

<a href="#null" onlick="$('#myiframe').attr('src','{ {=URL (....) } }');">click me to refresh iframe</a>

Date: Thu, 7 May 2009 13:56:23 -0700 (PDT)

import xlrd

path=os.path.join(request.folder,'private','a_dummy_file_name.xls')

open(path,'w').write(request.vars.filename.file.read())

book = xlrd.open_workbook(path)

result['The number of worksheets is']=book.nsheets

result["Worksheet name(s)"]= book.sheet_names()

sh = book.sheet_by_index(0)

From http://www.web2py.com/AlterEgo/default/show/217
A treeview could be recursive built using the DOM:
Functions in templates
Web2py also provides an alternative way to build recursive structures using exclusively the template language. In fact it is possible to define functions in the template language that mix code and html. Here is an example:
{ {def build(item):} }
{ {if isinstance(item.list):} }
<ul>{ {for element in item:} }<li>{ {build(element) } }</li>{ {pass} }</ul>
{ {else:} }
{ {=item} }
{ {pass} }
{ {return} }

Date: Sun, 10 May 2009 15:31:06 -0700 (PDT)

db((db.mydatefield>datetime.date(1998,12,31))?(db.mytabefield<datetime.date(2000,1,1))).select()

From http://www.web2py.com/AlterEgo/default/show/156

On app engine I add this to app.yaml, so static requests do not touch the app server:

- url: /([^/]+)/static/([^/]+)
static_dir: applications/\1/static/\2

Date: Thu, 5 Mar 2009 17:35:48 -0800 (PST)

db.define_table("TreeStructure",

SQLField("name", "string",notnull=True, default=None),

SQLField("parentID", "reference TreeStructure"))

Date: Thu, 12 Mar 2009 07:46:49 -0700 (PDT)

db.TreeStructure.parentID.requires=IS_NULL_OR(IS_IN_DB(db, 'TreeStructure.id'))

http://groups.google.com/group/web2py/browse_thread/thread/19ba7b69cde0a93dform.components[0].append(myfield)

web2py Widget Overview

Web2py contains a base web control layer called HTML helpers in file gluon/html.py
Html.py also contains class FORM. Based on these HTML helper classes is a collection of SQLFORM - specific widget classes layered over the HTML helper classes.

The widgets in web2py are
       class IntegerWidget(StringWidget)
       class DoubleWidget(StringWidget)
       class TimeWidget(StringWidget)
       class DateWidget(StringWidget)
       class DatetimeWidget(StringWidget)
       class TextWidget
       class BooleanWidget
       class OptionsWidget
       class MultipleOptionsWidget
       class PasswordWidget
       class UploadWidget

The widget classes are in source file gluon/sqlhtml.py which bundles them with the SQLFORM class.

So "widget" in this context really means "sql widget".

The SQLFORM class derives from HTML helper class FORM.
The HTML helpers are similar to ASP.NET web controls in that the HTML helper constructor emits HTML markup.
This happens because each HTML Helper class is derived from the DIV class and the
__init__ method of DIV emits the markup.

Here are a few more parameters I found mentioned in the forum:

SQLFORM(...,showid=False)

Architecture of running Web2py applications on GAE

One surprising aspect of running web2py apps on GAE is that you must deploy the web2py engine in its source code form!! So GAE runs web2py which in turns runs the web2py applications under it. GAE compiles the .py files into byte code on startup and it remains cached in GAE for 60 seconds. A good trick for performance would be to implement a CRON job under GAE which will ping one of your web2py controller files every 60 seconds in order to keep its bytecode fresh in the GAE cache. That potential trick is discussed here

http://groups.google.com/group/web2py/browse_thread/thread/ab55a1fa3c3e5ab5#

By the way, the standard 'Welcome' application the comes with web2py actually performs exceptionally poorly just because of its heavy use of Javascript animation (I believe) and so if one were to use it to benchmark web2py response times, one would get a misleading impression that it is very slow. If you run a web2py app which has a typical form, and put it on GAE and hit refresh repeatedly, it will respond quite quickly. Try refreshing this page for example. This is web2py on GAE in action.

There is a new release 1.2.2 of the GAE SDK which fixes some serious query bugs
with != and also sort order bugs.

And it adds a new feature to allow searching on the INDEX only.

Massimo is refactoring the DAL in about 2 weeks (mid June 2009) at which time it will be more customizable. And further DAL refactoring is promised soon after that which will have more GAE-specific datastore support.

Google WAVE on GAE

The recent Google Chat mentioned that WAVE and GAE can work together already.

Using JQuery in web2py

web2py files using JQuery were recently changed to use 'JQuery' instead of $

If the jquery.js is the very first include, then that will work.

What Javascript WYSIWYG HTML Editor to use

TinyMCE seems to be the best, most robust. NicEdit does not work properly with CHROME. The icons disappear.
NicEdit also loses all the <script> tags when you save from HTML edit mode.

What IDE / Debugger to use for web2py / python development

WING IDE is great. You can try it free for a month, and free 10 minute emergency sessions thereafter.