Introduction to Web Development – The Extended Web Stack December 5, 2011Posted by ficial in brain dump, techy.
Web development (http://en.wikipedia.org/wiki/Web_development) is often framed in terms of layers, levels, or a stack. This arises from the abstract and technical way the application is sliced. The top-most layer is one closest to the end user – the person who is actually using the application to solve some particular problem or fulfill some particular need. The bottom layer is the one conceptually, and usually physically, is farthest from the end user – the operating system of the computer on which the application lives. The full stack is, from the bottom up, the operating system, the database engine, the database schema, the web server software, the server-side programming language, the application framework (optional), the server-side business logic, the client-side programming language, the presentation/UI language, the user experience/design, and the client-side business logic. Each layer depends on the ones below it and supports the ones above it. The overall application idea spans the whole stack. Web developers tend to specialize in a few closely related layers, but benefit greatly from a working knowledge of the whole stack. Web developers also need to understand more general programming concepts such as client-server architecture, object oriented programming, documentation, code style, debugging, event-driven systems, logic, and algorithms. Learning and using a source control system is also very important – git (http://en.wikipedia.org/wiki/Git_(software)) is a good one these days.
A web application starts with an over-arching idea – a problem to be solved or a service to be provided or some such thing. This informs work at all layers. It is totally independent of the specific technology used, to the point where it might even be implemented entirely without a computer. Before any of the detailed work begins one must have a thorough understanding the the application idea. I’ll go through rest of the layers from top to bottom, but that is NOT necessarily the order in which they’re implemented – the software development process is related to but separate from the software organization.
At the very top is the client-side business logic layer. This is the most important layer because it defines what the end user is actually allowed to do and how. This layer often has no direct technical representation. When it does, it appears in various forms of automated test tools/routines. The conceptual tools for this level are use cases (http://en.wikipedia.org/wiki/Use_case), user stories (http://en.wikipedia.org/wiki/User_story), and state and flow diagrams (http://en.wikipedia.org/wiki/State_diagram). The implementation work is usually done in a word processing program. Complicated systems may require / use UML (http://en.wikipedia.org/wiki/Unified_Modeling_Language). There aren’t very many automated test tools for client side business logic, so if you end up dealing with one you’ll almost certainly have to learn something specific to that project. Key skills at this layer are analyzing, hypothesizing, organizing, documenting, and proficiency and any technical tools that are used. The finished product of work at this layer is a clear, concise, and formal understanding of what the user is and isn’t allowed to do (and ideally at least a reference to why).
Next is the user interface/experience(UI/UX)/design layer (http://en.wikipedia.org/wiki/User_interface, http://en.wikipedia.org/wiki/User_experience). This is the most important layer because it determines whether your application will actually be used by and useful for the end user – the difference between a failed project and a successful one is not in the idea but in whether or not people want to use it. Conceptual work at this layer is done using tools like card sorts (http://en.wikipedia.org/wiki/Card_sorting, http://www.measuringux.com/CardSorting/, user surveys, flow diagrams (http://en.wikipedia.org/wiki/Control_flow_diagram), concept maps (http://en.wikipedia.org/wiki/Concept_map), and lots of rough sketches (usually in pen or pencil on paper). The goal of the conceptual work is to figure out how to present the underlying business logic in a way that is intuitive and appealing. The implementation work here is usually done in a graphics design program – adobe photoshop, adobe illustrator, or the open source package GIMP are all good options. There are also some good programs available for doing the rougher, faster work – http://mockflow.com/ is a good one. Key skills at this layer are user psychology, graphic design, visualization, artistit creation/vision, and technical skills with whatever graphics packages/applications are used. The finished product of work at this layer is usually design mockups (images show what components of the applications look like) and wireframes (low/no graphic representations of how a person uses and navigates the application).
Next is the server side business logic. This is the most important layer because it defines what your application does; it provides the why’s for everything that your application does. The tools used for this layer are the same as for the client-side business logic layer. The focus at this layer tends to be less on users and more on data and events (though the data and events often originate from users). At this layer one has to define both support for user operations/actions and non-user-dependent, system-wide logic. The conceptual boundary between the client-side and server-side business logic is fuzzy, though in the end it is implemented in one place or the other. Security and privacy considerations (http://www.taoti.com/blog_web_security.php, http://code.google.com/edu/security/index.html, http://en.wikipedia.org/wiki/Web_development#Security_Considerations) will usually be major factors in deciding where a piece of logic finally resides. Key skills at this layer are analyzing, hypothesizing, organizing, documenting, and proficiency and any technical tools that are used. The finished product of work at this layer is a clear, concise, and formal understanding of what the application does.
Next is the application framework. This layer is optional – some applications depend heavily on it, and others eschew it entirely. The role of a framework is something of a matter of opinion and style – different developers will give wildly varying answers as to which framework should be used to what degree for any given application. An application framework is a set of tools and template code in a given organization model which together assist in the the creation of a web application. There are specialized frameworks (such as WordPress (http://wordpress.org/) or Drupal (http://drupal.org/) for content management, or MediaWiki (http://www.mediawiki.org/wiki/MediaWiki) or Twiki (http://twiki.org/) for wikis) and general frameworks (e.g. Rails (http://rubyonrails.org/), Zend (http://framework.zend.com/), Cake (http://cakephp.org/), Symfony (http://www.symfony-project.org/), and JSP (http://www.oracle.com/technetwork/java/javaee/jsp/index.html)). The conceptual tools vary from one framework to the next. If you want or have to use a given framework start by reading its high level documentation – you should get an understanding of the general approach that the framework uses and the high-level what it organizes and implements an application. The implementation tools are the language(s) in which the framework is written, the API/library code the framework provides, and the development tools the framework provides.
Next is the server side programming language. This is the most important layer because it makes everything on the server actually work. If an application framework used it will define also which language is used. Server side programming languages for web development are general, powerful programming languages that usually are especially adept at string manipulation and handling abstract data objects. Often they are ‘scripting languages’, which connotes looser style and faster development, but server-side code could certainly be written in a more formal, structured language. Some common server-side web programming languages are PHP (http://php.net/manual/en/index.php), Python (http://python.org/), Ruby (http://www.ruby-lang.org/en/), Perl (http://www.perl.org/), and Java (http://en.wikipedia.org/wiki/Java_(programming_language)). The conceptual tools at this layer are all general programming ones. The technical tools are text editors and IDEs. Key skills for this layer are all the general programming techniques, the specifics of whatever language is used, and usually some associated system tools (e.g. compilers). The finished product of this layer is a set of files containing code and documentation.
Next is the database schema. This is the most important layer because it defines the data that your application uses and stores. Historically this has almost always been a relational database (http://en.wikipedia.org/wiki/Relational_database), but in recent times non-relational databases (e.g. http://en.wikipedia.org/wiki/NoSQL) have become more popular. The conceptual tools at this layer include data modeling (http://en.wikipedia.org/wiki/Data_modeling), entity-relationship diagrams (ERD – http://en.wikipedia.org/wiki/Entity_relationship_model), normalization (http://en.wikipedia.org/wiki/Database_normalization), and data abstraction. The implementation tools include data modeling tools (http://stackoverflow.com/questions/166557/a-good-database-modeling-tool), text editors, engine specific utilities, and possibly framework-specific tools/utilities. Key skills for this layer are data modeling and the the technical skill of SQL (http://en.wikipedia.org/wiki/SQL). This finished product of this layer is a precise definition of the data that the application stores, and usually how each piece is related to others. Often this is in the form of files containing SQL.
Next is the database engine. This is the most important layer because it make the database schema actually work. Most engines will support most schemas, so the choices here tend to depend on budget and any specialized features required. The most popular open source database engines are MySQL (http://www.mysql.com/) and Postgres (http://www.postgresql.org/). The latter is slightly more technically tricky but has slightly more advanced features. The most common commercial options are Microsoft SQLServer and Oracle DB. Unless the web development work is for a large company, one of the open source packages is most likely, and in general MySQL is more likely than Postgres. Key skills for this level include basic operating system skills (at least installing and configuring software), DB system administration (e.g. how to implement a schema, how to back-up and restore a set of data, etc.) and use of whatever utilities a given engine has available (usually command-line tools or GUI for the open source engines and GUI tools for the commercial ones). Knowledge of SQL is also very important for this layer. The finished product of this layer is a system that fetches, stores, maintains, and controls access to the data defined in the schema.
Next is the web server software (http://en.wikipedia.org/wiki/Web_server). This is the most important layer because it allows communication between the client and the server – it puts the ‘web’ in ‘web application’. From the web developer’s perspective there are three especially important aspects of web server software: configuration, tuning, and logging. Configuration is basically the process of getting it running at all – turning code on the server into web pages on the client. Tuning is the process of getting it to run well – making the application fast and reliable with as few resources as possible. Logging is tracking information about how the application is being used, and is especially important in debugging and performance monitoring. The most common and popular web server is apache (http://httpd.apache.org/), with nginx (engine-x – http://wiki.nginx.org/Main) coming into common usage for specialized, simple needs. Microsoft Internet Information Server (IIS) is used by some larger companies. Various languages and frameworks require particular additional modules or settings, but such things are generally well documented. Key skills for this level include basic operating system skills (at least installing and configuring software), and the technical skills specific to whatever software package is used. The finished product of this layer is the ability to view web pages served from that computer. NOTE: for a development system the computer on which you work is often the same as the one that’s running the webserver – that is, the same machine is acting as both host and client. This is fine as long as one access web pages via the server rather than the file system.
Finally, at the very bottom, is the operating system of the web host (‘host’ refers to the machine, to distinguish it from the server, which in general parlance can mean either the machine or the software). This is the most important layer because with out a web host nothing on the server side exists. This layer really subsumes a whole bunch of other, even lower level functions like network connectivity and firewalling and that kind of thing. However, a web developer rarely needs to know anything about that level (that’s all in the dark and mysterious realm of the network and system administrators). The OS level is usually of concern for the web developer in setting up and maintaining their development environment – the production environment is typically managed my system administrator specialists. Key concepts and skills of this level include system security, user management, persistence (backup and restore), software installation, update management, and configuration. Linux (http://en.wikipedia.org/wiki/Linux, http://en.wikipedia.org/wiki/Linux_distribution) is a good choice for an OS for web development (Mint (http://linuxmint.com/) and CentOS (http://www.centos.org/) are good versions/distributions), though work can certainly be done on Windows or Mac as well. Many inexpensive web hosting services offer a command line option, which is access to the operating system level. The finished product of this layer is a machine that runs and that one can log in to and connect to the internet with.
When developing a web application the layers are tackled in a different order, and since they’re interdependent usually work progresses on multiple layers at once. When developing the order is something like:
1. application idea
2. server business logic; client business logic
3. client UI/UX design; web host; web server; database engine; set up development environment(s)
4. database schema (and test/sample data)
5. client UI/UX implementation; server side implementation