Developer Notes: Drupal, Hudson, and Mercurial Auto build

The following are notes to my developers which I share now for the benefit of clients and dev teams who are curious how an enterprise build system may be set up.

Understanding the repo structure

The Mercurial repos in Kiln are structured as follows:

Project->Environment Stack [repo group] -> Repos (code, dbsnaps, [others])

Project

Projects are usually titled with the project number preceded by the letter “p”. The letter “p” is short for “project” and is only there because Kiln (and other repos) will not accept just numbers for a project title.

In a few cases, like for the RRA website, the project will have a string (text) name instead of a project number. This is because multiple concurrent projects are running for the RRA web site and it makes sense to keep the work under one project. In this case the project is titled, “prra”. Note the letters are all lower case.

Environment Stack

The default stack is called, “main”. This is the dev/integration environment on the cloud dev server.

Other stacks may include qa, loadtesting, production, or anything else needed. These stacks may exist on the dev server, or other servers, including the production server.

7-5-2010 1-28-13 PM

Repos

Change Note: This section notes our change from the practice of separating Drupal code into three repos. We have found the benefits of managing all code in one repo outweighs longer initial pull or push times. The older version of this section remains at the end of this doc for historical purposes.

code

This repo combines the previous three repos (drupalcore, sites, and sitesall) into one. This repo contains all code to run the client site, as well as special files build scripts which can modify the build process at runtime.

dbsnaps

This repo may contain multiple database snapshots (sql files). The one file titled, “db.sql” is the primary source file for site builds.

Here is a typical scenario:

  1. We start a new dev project (new or existing site)

a. We receive an sql file which is a full sql dump of the site database (hopefully with the cache and watchdog tables truncated).

b. This file goes into the dbsnaps repo and will be titled, “db.sql”.

c. We should all build off this version controlled file.

d. Auto builds will also take place off this file.

  1. We may decide to test some changes to the branch we are working on.

a. These changes could involve changes to the data structure.

b. To help us roll back quickly, or to make a diff of the changes we made, we can use the backup_and_migrate module to output a snapshot before the changes and then output a snapshot after these changes.

c. These files may be checked into the repo for safe keeping and may be used to create a diff file/patch to look for exactly what changes were made to what tables. Remember, these sql files are not the primary source for the builds, and db.sql will overwrite the db on the next build.

  1. Actual db changes should be made in code via views export, features, chaos tools, and sql statements in hook_update functions.

How to trigger a build

To trigger a build, push a change to the repo.

That’s it.

If you’d like to know what happens after that, read on.

The Kiln/HG web hook will trigger a call to the dispatcher script on our dev server.

  1. The call to the dispatcher includes a token to authenticate its source

  2. and a json payload containing information about the check in.

The dispatcher will then:

  1. authenticate the source of the call and record the call in its log file

  2. upon authentication prepare the secure call to Hudson to trigger the build with parameters parsed and passed from the json payload

  3. ignore a call to build on the production server as these are manually controlled via Hudson

  4. perform miscellaneous tasks such as emailing notifications and adding a record of the dispatch to our intranet

Once Hudson receives the call to start a build, the following happens:

  1. The repo is initially cloned if it does not exist on the dev server, or pulled from then updated with a hg update –clean.

  2. A build ini file is searched for in the updated repo and if found parsed for variables.

  3. A prebuild sh script is searched for in the updated repo and if found ran.

  4. The project is then build with an import of the dbsnap and a symlink from the web root to the code repo.

  5. A build sh script is searched for in the updated repo and if found ran.

  6. A vhost file is added to /etc/apache2/sites-available/ in Apache, enabled, and Apache is reloaded.

  7. A postbuild sh script is searched for in the updated repo and if found ran.

  8. The following Drush and shell commands are ran in downstream jobs and the results recorded in the Hudson console and actions are triggered by results.

Miscellaneous cleanup and logging

a. drush –yes pm-enable dblog

b. drush –yes watchdog-delete all

c. drush –yes cache-clear all

d. drush –yes updatedb

e. drush –yes cron

f. drush –yes pm-refresh

g. drush –yes status

h. drush –yes watchdog-show

i. drush –yes pm-list

Run Coder Module from Drush

j. drush –yes dl coder-6.x-2.x-dev

k. drush –yes pm-enable coder

l. drush –yes coder contrib critical

Patch and run Simpletests

m. drush –yes dl simpletest-6.x-2.x-dev

n. patch -f -p0 < ${WWWROOT}${STACK}${PROJECT}/sites/all/modules/simpletest/D6-core-simpletest.patch

o. wget http://drupal.org/files/issues/simpletest_run_on_any_profile_6.16_0.patch

p. patch -f -p0 < ${WWWROOT}${STACK}${PROJECT}/sites/all/modules/simpletest/simpletest_run_on_any_profile_6.16_0.patch

q. cd ${WWWROOT}${STACK}${PROJECT}

r. drush –yes pm-enable simpletest

s. mv ${WWWROOT}${STACK}${PROJECT}/sites/all/modules/simpletest/run-tests.sh ${WWWROOT}${STACK}${PROJECT}/scripts

t. rm -rf ${WWWROOT}${STACK}${PROJECT}/scripts/tests/*.xml

u. /usr/bin/php ${WWWROOT}${STACK}${PROJECT}/scripts/run-tests.sh –url http://${DEVURL}/${STACK}${PROJECT} –all –color

Tools

  1. Drush: http://drupal.org/project/drush

  2. Coder: http://drupal.org/project/coder

  3. Simpletest: http://drupal.org/simpletest

 

Deprecated Section on Repos

We use four repos to break code into sets that are managed differently and/or updated at different frequencies. This separation of code into sets speeds up your most frequent pulls and pushes and our most frequent auto builds.

drupalcore

The “drupalcore” repo is updated least often. It should contain only core Drupal code and should be unchanged from the version in which it was downloaded from Drupal.org. For example: the Drupal 6.16 zip file. Sometimes we may receive client code which has Drupal core code modified. We should try to bring the client’s code back to core by moving any hacks on core into a custom module or theme function.

sites

The “sites” repo contains everything in the sites folder found in a normal Drupal setup, except the “sites/all” directory. Additionally, ff the web site uses “sites/default” as its Drupal site folder then this directory will contain the site’s assets.

For the auto build process, the “sites” directory will have an extra site folder to be used on the dev server. For example, if the project number is 957, then the sites directory will contain a folder called, “main957.seanbuscay.net”. In “main957.seanbuscay.net” there will be a settings.php file with the config info needed to run on the development server.

This repo is rarely changed. We may, from time to time, add files to it from the production server so we may see the latest images and assets while testing on the dev server. No primary code changes should take place here.

sitesall

The structure of this repo is a little different from the others. It starts off with just one immediate child directory called, “all”. This folder corresponds to the typical Drupal setup path of “[Drupal root]/sites/all”. Inside this “all” folder are all modules (custom or contrib.), themes, libraries, and features.

Typically, this is where we do most of our coding. Adding modules or writing custom modules. Customizing themes and creating new ones. This repo is updated most often.

Please note: In some cases, a module or theme should only be exposed to one site instance in a Drupal multisite setup. If this is the case, we can either do our dev work in this repo and then later move the modules or code to the site instance folder, or start off in that site instance folder in the “sites” repo. In this case the “sites” repo will become our most frequently updated repo.

dbsnaps

This repo may contain multiple database snapshots (sql files). The one file titled, “db.sql” is the primary source file for site builds.

Share or Comment via Twitter

dev ops, drupal, drush, testing, web development