A web application starts with an over-arching idea – a problem to be solved or a service to be provided or some such thing. This informs work at all layers. It is totally independent of the specific technology used, to the point where it might even be implemented entirely without a computer. Before any of the detailed work begins one must have a thorough understanding the the application idea. I’ll go through rest of the layers from top to bottom, but that is NOT necessarily the order in which they’re implemented – the software development process is related to but separate from the software organization.

At the very top is the client-side business logic layer. This is the most important layer because it defines what the end user is actually allowed to do and how. This layer often has no direct technical representation. When it does, it appears in various forms of automated test tools/routines. The conceptual tools for this level are use cases (http://en.wikipedia.org/wiki/Use_case), user stories (http://en.wikipedia.org/wiki/User_story), and state and flow diagrams (http://en.wikipedia.org/wiki/State_diagram). The implementation work is usually done in a word processing program. Complicated systems may require / use UML (http://en.wikipedia.org/wiki/Unified_Modeling_Language). There aren’t very many automated test tools for client side business logic, so if you end up dealing with one you’ll almost certainly have to learn something specific to that project. Key skills at this layer are analyzing, hypothesizing, organizing, documenting, and proficiency and any technical tools that are used. The finished product of work at this layer is a clear, concise, and formal understanding of what the user is and isn’t allowed to do (and ideally at least a reference to why).

Next is the user interface/experience(UI/UX)/design layer (http://en.wikipedia.org/wiki/User_interface, http://en.wikipedia.org/wiki/User_experience). This is the most important layer because it determines whether your application will actually be used by and useful for the end user – the difference between a failed project and a successful one is not in the idea but in whether or not people want to use it. Conceptual work at this layer is done using tools like card sorts (http://en.wikipedia.org/wiki/Card_sorting, http://www.measuringux.com/CardSorting/, user surveys, flow diagrams (http://en.wikipedia.org/wiki/Control_flow_diagram), concept maps (http://en.wikipedia.org/wiki/Concept_map), and lots of rough sketches (usually in pen or pencil on paper). The goal of the conceptual work is to figure out how to present the underlying business logic in a way that is intuitive and appealing. The implementation work here is usually done in a graphics design program – adobe photoshop, adobe illustrator, or the open source package GIMP are all good options. There are also some good programs available for doing the rougher, faster work – http://mockflow.com/ is a good one. Key skills at this layer are user psychology, graphic design, visualization, artistit creation/vision, and technical skills with whatever graphics packages/applications are used. The finished product of work at this layer is usually design mockups (images show what components of the applications look like) and wireframes (low/no graphic representations of how a person uses and navigates the application).

Next is the presentation/UI language. This is the most important layer because it allows a user to actually interact with the application. In the wider programming world there are many tools used for this layer, but in the web world it’s much more limited (by the fact that there is a web browser running all this). HTML, CSS, and Javascript are the fundamental pieces of this layer. Two absolutely key concepts that bind those three together are the document object model (DOM – http://en.wikipedia.org/wiki/Document_Object_Model) and event handling (http://en.wikipedia.org/wiki/Event_handling). At this level Javascript is usually as a part of some sort of framework such as jQuery (http://en.wikipedia.org/wiki/Jquery, http://jquery.com/). HTML and CSS serve double roles here. In the first role, they mark up text and other data and tell the browser how to display it. In the second role they provide handles which the Javascript uses to manipulate things. Conceptual tools of this layer include event diagrams, flow diagrams, and general programming tools (e.g. iterative refinement, stubbing, problem abstraction, refactoring, etc.) Implementation tools of this level are a web browser (or browsers, for testing across multiple platforms), and some kind of programming environment. The latter may be a simple text editor or a full-blown IDE or anything in between. Key skills at this layer are program/code design, logic, visualization, documentation and technical skills with HTML, CSS, Javascript, and whatever Javascript framework is used. The finished product of work at this layer is something a user can interact with in a browser.

Next is the client side programming language. This is the most important layer because with out it the client side does nothing. This is tightly entwined with the presentation/UI language. In web development this is pretty much always going to be Javascript (http://en.wikipedia.org/wiki/JavaScript, http://www.w3schools.com/js/). The distinction between this level and the previous one is that this is more general whereas the previous one is specifically about presentation and interface management. In many ways the framework used above hides the richness and complexity of the client-side language as a whole. This level divorces the data manipulation from the user interaction – at this layer one is only concerned with code and data. Key skills for this layer are all the general programming techniques, and the specifics of the Javascript language. The finished product of this layer is a set of files containing Javascript code and documentation.

Next is the server side business logic. This is the most important layer because it defines what your application does; it provides the why’s for everything that your application does. The tools used for this layer are the same as for the client-side business logic layer. The focus at this layer tends to be less on users and more on data and events (though the data and events often originate from users). At this layer one has to define both support for user operations/actions and non-user-dependent, system-wide logic. The conceptual boundary between the client-side and server-side business logic is fuzzy, though in the end it is implemented in one place or the other. Security and privacy considerations (http://www.taoti.com/blog_web_security.php, http://code.google.com/edu/security/index.html, http://en.wikipedia.org/wiki/Web_development#Security_Considerations) will usually be major factors in deciding where a piece of logic finally resides. Key skills at this layer are analyzing, hypothesizing, organizing, documenting, and proficiency and any technical tools that are used. The finished product of work at this layer is a clear, concise, and formal understanding of what the application does.

Next is the application framework. This layer is optional – some applications depend heavily on it, and others eschew it entirely. The role of a framework is something of a matter of opinion and style – different developers will give wildly varying answers as to which framework should be used to what degree for any given application. An application framework is a set of tools and template code in a given organization model which together assist in the the creation of a web application. There are specialized frameworks (such as WordPress (http://wordpress.org/) or Drupal (http://drupal.org/) for content management, or MediaWiki (http://www.mediawiki.org/wiki/MediaWiki) or Twiki (http://twiki.org/) for wikis) and general frameworks (e.g. Rails (http://rubyonrails.org/), Zend (http://framework.zend.com/), Cake (http://cakephp.org/), Symfony (http://www.symfony-project.org/), and JSP (http://www.oracle.com/technetwork/java/javaee/jsp/index.html)). The conceptual tools vary from one framework to the next. If you want or have to use a given framework start by reading its high level documentation – you should get an understanding of the general approach that the framework uses and the high-level what it organizes and implements an application. The implementation tools are the language(s) in which the framework is written, the API/library code the framework provides, and the development tools the framework provides.

Next is the server side programming language. This is the most important layer because it makes everything on the server actually work. If an application framework used it will define also which language is used. Server side programming languages for web development are general, powerful programming languages that usually are especially adept at string manipulation and handling abstract data objects. Often they are ‘scripting languages’, which connotes looser style and faster development, but server-side code could certainly be written in a more formal, structured language. Some common server-side web programming languages are PHP (http://php.net/manual/en/index.php), Python (http://python.org/), Ruby (http://www.ruby-lang.org/en/), Perl (http://www.perl.org/), and Java (http://en.wikipedia.org/wiki/Java_(programming_language)). The conceptual tools at this layer are all general programming ones. The technical tools are text editors and IDEs. Key skills for this layer are all the general programming techniques, the specifics of whatever language is used, and usually some associated system tools (e.g. compilers). The finished product of this layer is a set of files containing code and documentation.

Next is the database schema. This is the most important layer because it defines the data that your application uses and stores. Historically this has almost always been a relational database (http://en.wikipedia.org/wiki/Relational_database), but in recent times non-relational databases (e.g. http://en.wikipedia.org/wiki/NoSQL) have become more popular. The conceptual tools at this layer include data modeling (http://en.wikipedia.org/wiki/Data_modeling), entity-relationship diagrams (ERD – http://en.wikipedia.org/wiki/Entity_relationship_model), normalization (http://en.wikipedia.org/wiki/Database_normalization), and data abstraction. The implementation tools include data modeling tools (http://stackoverflow.com/questions/166557/a-good-database-modeling-tool), text editors, engine specific utilities, and possibly framework-specific tools/utilities. Key skills for this layer are data modeling and the the technical skill of SQL (http://en.wikipedia.org/wiki/SQL). This finished product of this layer is a precise definition of the data that the application stores, and usually how each piece is related to others. Often this is in the form of files containing SQL.

Next is the database engine. This is the most important layer because it make the database schema actually work. Most engines will support most schemas, so the choices here tend to depend on budget and any specialized features required. The most popular open source database engines are MySQL (http://www.mysql.com/) and Postgres (http://www.postgresql.org/). The latter is slightly more technically tricky but has slightly more advanced features. The most common commercial options are Microsoft SQLServer and Oracle DB. Unless the web development work is for a large company, one of the open source packages is most likely, and in general MySQL is more likely than Postgres. Key skills for this level include basic operating system skills (at least installing and configuring software), DB system administration (e.g. how to implement a schema, how to back-up and restore a set of data, etc.) and use of whatever utilities a given engine has available (usually command-line tools or GUI for the open source engines and GUI tools for the commercial ones). Knowledge of SQL is also very important for this layer. The finished product of this layer is a system that fetches, stores, maintains, and controls access to the data defined in the schema.

Next is the web server software (http://en.wikipedia.org/wiki/Web_server). This is the most important layer because it allows communication between the client and the server – it puts the ‘web’ in ‘web application’. From the web developer’s perspective there are three especially important aspects of web server software: configuration, tuning, and logging. Configuration is basically the process of getting it running at all – turning code on the server into web pages on the client. Tuning is the process of getting it to run well – making the application fast and reliable with as few resources as possible. Logging is tracking information about how the application is being used, and is especially important in debugging and performance monitoring. The most common and popular web server is apache (http://httpd.apache.org/), with nginx (engine-x – http://wiki.nginx.org/Main) coming into common usage for specialized, simple needs. Microsoft Internet Information Server (IIS) is used by some larger companies. Various languages and frameworks require particular additional modules or settings, but such things are generally well documented. Key skills for this level include basic operating system skills (at least installing and configuring software), and the technical skills specific to whatever software package is used. The finished product of this layer is the ability to view web pages served from that computer. NOTE: for a development system the computer on which you work is often the same as the one that’s running the webserver – that is, the same machine is acting as both host and client. This is fine as long as one access web pages via the server rather than the file system.

Finally, at the very bottom, is the operating system of the web host (‘host’ refers to the machine, to distinguish it from the server, which in general parlance can mean either the machine or the software). This is the most important layer because with out a web host nothing on the server side exists. This layer really subsumes a whole bunch of other, even lower level functions like network connectivity and firewalling and that kind of thing. However, a web developer rarely needs to know anything about that level (that’s all in the dark and mysterious realm of the network and system administrators). The OS level is usually of concern for the web developer in setting up and maintaining their development environment – the production environment is typically managed my system administrator specialists. Key concepts and skills of this level include system security, user management, persistence (backup and restore), software installation, update management, and configuration. Linux (http://en.wikipedia.org/wiki/Linux, http://en.wikipedia.org/wiki/Linux_distribution) is a good choice for an OS for web development (Mint (http://linuxmint.com/) and CentOS (http://www.centos.org/) are good versions/distributions), though work can certainly be done on Windows or Mac as well. Many inexpensive web hosting services offer a command line option, which is access to the operating system level. The finished product of this layer is a machine that runs and that one can log in to and connect to the internet with.

When developing a web application the layers are tackled in a different order, and since they’re interdependent usually work progresses on multiple layers at once. When developing the order is something like:
1. application idea
2. server business logic; client business logic
3. client UI/UX design; web host; web server; database engine; set up development environment(s)
4. database schema (and test/sample data)
5. client UI/UX implementation; server side implementation

My Gemfile has:

gem 'pg'

and there weren’t any particular issues I ran into there (just the usual bundle install). The tricky parts were the database.yml settings, and getting postgres itself running. For the former I did a bit of googling and found a great set of examples at https://gist.github.com/961978. Adapting the postgres example was pretty easy. My database.yml looks like:

development:
 adapter: postgresql
 encoding: unicode
 database: proj_development
 pool: 5
 username: projuser
 password: 12345dev # you may want something more secure here :P
 timeout: 5000

# Warning: The database defined as "test" will be erased and
 # re-generated from your development database when you run "rake".
 # Do not set this db to the same as development or production.
 test:
 adapter: postgresql
 encoding: unicode
 database: proj_test
 pool: 5
 username: projuser
 password: 12345test # you may want something more secure here :P
 timeout: 5000

production:
 adapter: postgresql
 encoding: unicode
 database: proj
 pool: 5
 username: projuser
 password: 12345prod # you may want something more secure here :P
 timeout: 5000

Getting postgres running took a quite a bit more research and experimenting. The difficulty I ran into was that much of the documentation out there is for systems other than cygwin, and cygwin has a number of idiosyncracies with respect to where various files live and that kind of thing. Two pages I found to be useful were http://beige.ucs.indiana.edu/I590/node149.html and http://www.postgresql.org/docs/8.4/static/installation-platform-notes.html. In addition to those I made heavy use of man and whereis to figure out where various commands resided and what exactly they did. The eventual sequence of successful (and relevant commands) was:

$ /usr/sbin/cygserver &
 cygserver: Initialization complete. Waiting for requests.
$ cd
$ mkdir pgdata
$ cd pgdata
$ /usr/sbin/initdb .
 The files belonging to this database system will be owned by user "Chris".
 This user must also own the server process.
The database cluster will be initialized with locale C.UTF-8.
 The default database encoding has accordingly been set to UTF8.
fixing permissions on existing directory . ... ok
 creating subdirectories ... ok
 selecting default max_connections ... 40
 selecting default shared_buffers/max_fsm_pages ... 32MB/204800
 creating configuration files ... ok
 creating template1 database in ./base/1 ... ok
 initializing pg_authid ... ok
 initializing dependencies ... ok
 creating system views ... ok
 loading system objects' descriptions ... ok
 creating conversions ... ok
 setting privileges on built-in objects ... ok
 creating information schema ... ok
 vacuuming database template1 ... ok
 copying template1 to template0 ... ok
 copying template1 to postgres ... ok
WARNING: enabling "trust" authentication for local connections
 You can change this by editing pg_hba.conf or using the -A option the
 next time you run initdb.
Success. You can now start the database server using:
/usr/sbin/postgres -D .
 or
 /usr/sbin/pg_ctl -D . -l logfile start
$ createdb proj_development
 CREATE DATABASE
$ /usr/sbin/createuser -P projuser
 Enter password for new role:
 Enter it again:
 Shall the new role be a superuser? (y/n) n
 Shall the new role be allowed to create databases? (y/n) n
 Shall the new role be allowed to create more new roles? (y/n) n
 CREATE ROLE

Once all that was done I was able to run rake db:migrate (from my project folder, of course) to set up my tables, and then rails s to get my dev server going. Shockingly, those two steps went without a hitch, and I was able to use my application on my dev machine without any problem. Yay!

….except that heroku is still unhappy and gives the same ‘PGError: ERROR: operator does not exist: character varying = integer’ as I got originally. Dammit. :(

So, after re-examining all my migration files once more again from the start, I noticed this time that I’d some how used t.string instead of t.integer for a foreign key on an id field. Oops. I fixed that line, committed it, pushed to heroku, ran the db migration all the way down then back up, and POOF! IT WORKS!

Still, at least I learned something, and hopefully now you have as well.

The first area I explored was the expansion of resources to more generalized game state dimensions. In addition to explicit in-game resources the graph would include things like cards in hand, actions, and infrastructure. This quickly lead to two refinements.

I differentiated the vertexes into two sub-types. First were resources, explicit or implicit dimensions of the game state which are quantifiable. Second were actions, rules of the game that allowed the conversion of one resource to another. No resource vertex can be directly connected to another resource vertex, and no action vertex can be directly connected to another action vertex; edge paths alternate between resource and action vertexes.

Edges were also subdivided into two types. First were conversion edges, indicating that the resource of the initial vertex is consumed/reduced or that the resource of the terminal vertex is created/increased. Second were influence edges, which always have a resource as their initial vertex and an action as their terminal vertex. The resource of an influence edge is NOT altered. Resources the are the initial vertex of an influence edge are called infrastructure, distinguishing them from consumable resources (from now on called simply ‘resources’).

After trying to use this for analysis I found I needed a refinement – actions were split based on their inputs and outputs. E.g. in the simpler version there might be a trade action that has wood, cows, and gold as conversion precedents (direct predecessors), a market as an influence precedent, and wood, cows and gold as subsequents (direct successors). This would be useful in indicating that the trade action allows the conversion of one resource to another and that having a market affects that somehow, but the details are fuzzy. By splitting the generic trade action into separate actions (market and non-market for each transition) it became clear exactly how the game worked. That refinement also enables weighting each of the conversion edges. The weight indicates how much the initial vertex resource is reduced, or how much the terminal vertex resource is increased. This graph is the resource-infrastructure-action (RIA) flow diagram. It encapsulates how a game works in a vacuum, but does not take into consideration randomizers (though one can use expected values), nor player volition (though one can use a minimax framework). However, it can still be a useful tool.

The RIA flow encompasses the basic functionality of a resource flow diagram in that it illustrates how one in-game resource is converted to another, an eventually to the score/points ‘resource’. It allows us to quantify the efficiency of a path – take the total resource inputs, divide by the total outputs and the number of actions required. This path could be from resources to points, or merely from one resource to another. As a special case of a 2-edge long path, the efficiency of individual actions can also be analyzed – divide the inputs by the outputs.

On a slightly more subtle level, it simplifies analysis of the values of infrastructure. Consider two actions that have identical resources as inputs and outputs (though with different weights), but one also has an infrastructure as an input. The efficiency of the infrastructure is then the difference in the efficiency of those alternate action. This can be generalized to full paths – instead of two actions consider two paths from the same initial resources to points, one of which includes one or more infrastructure-enabled actions and the other of which does not.

An ‘engine’ in a game is an indefinitely repeatable series of actions that generates a net positive change in one or more resources. Building an engine is a key part of a successful strategy in many economic games. In an RIA flow, potential engines are easily identified as cycles in the graph. A cycle is an actual engine if the efficiency of the path is greater than 1 (a direct engine), or if it is equal to 1 but also has output leaves that are not in the cycle (a side-effect engine). The efficiency of an engine can be calculated as the output divided by the number of steps/actions to complete the cycle. In an infinite game the start-up costs of the engine (i.e. what needs to be done to establish the necessary infrastructure and/or gather the prerequisite resource set to prime it) are irrelevant compared to the output, but in a finite situation they come into consideration. One can calculate the expected start up time and and costs (the number of actions and resources needed), and the expected output over the remainder of the game to determine over all expected value of the engine. If the product of the engine is not directly in scoring, then one must also take into account the number of actions needed to convert the output to points.

priming actions = PA
priming resources = PR
engine cycle actions = CA
engine cycle resources gained = CR
actions needed to convert engine output into scoring = SA
efficiency with which engine output is converted to points = CE
efficiency with which PR could have been converted to points = PE
total expected actions in game = GA

((GA-PA-SA)/CA * CR * CE) – (PR*PE) = expected net contribution to final score

Calculations get more complicated if actions may be pipe-lined, but the principle remains the same. There may be other useful digraph analytic tools that could be applied to an RIA flow diagram.

There are a number of fixes discussed there. The one that finally worked for me was to run
ln -nfs /usr/local/bin/git /usr/bin/git
since apparently where CentOS puts git and where the passenger code expects to find it are different. With that, everything worked fine.

I cannot decide whether it’s frustrating or gratifying to fix hours worth of stymieing with a single, simple command.

I had done some research on this for my old system, and found quite a bit of discussion about this problem and how to solve it (e.g. http://www.garethhunt.com/2008/02/11/cygwin-died-waiting-for-dll-loading/, http://stackoverflow.com/questions/4988091/unable-to-start-solr-server-ruby-on-rails). Sadly, that approach (quit all cygwin shells, run c:\cygwin\bin\ash.exe, $ /bin/rebaseall) did not work for me then, and still wasn’t working on the new system. On my old machine I just ignored the problem since it wasn’t a blocker, but on my new system it’s really in the way, so I dove into it a bit more. I eventually came across this really helpful message by Illia Bobyr: http://cygwin.com/ml/cygwin/2011-04/msg00075.html.

In brief, the problem is that rebaseall doesn’t necessarily know all the .dll’s and .so’s it has to fix. The solution is to compose your own list and run rebaseall against that list using the -T option. For me, the process that finally worked was:

1. in a cygwin shell: find /bin /lib /usr -iname '*.so' > /tmp/to_rebase.lst
2. in a cygwin shell: find /bin /lib /usr -iname '*.dll' >> /tmp/to_rebase.lst
3. quit all cygwin shells and stop all cygwin processes
4. run as Administrator C:\cygwin\bin\ash.exe
5. /bin/rebaseall -T /tmp/to_rebase.lst
6. exit ash
7. restart machine

And now, finally, I can code in peace….

Until the next problem appears.