Neogregious

Migrating a Subversion Project to Git on BitBucket

2011-10-18T20:57:00.000-07:00

Subversion is still the gold standard for version control (a.k.a. source control) systems, but the new kid on the block - Git - has undeniable advantages for many development models. Even for solo or centralised hierarchical development organisations, Git's agile branching and merging make it a joy to use.

But if you're going to go to the effort of switching version control systems, you probably also want to consider moving your repository to the cloud. This increases your options for off-site developers, and reduces the risks and maintenance costs of internally managed version control servers.

This article describes how to migrate a Subversion repository to Bitbucket, one of several online repositories.
First of all, why Bitbucket? GitHub and Google Code are worthy alternatives, but both have restrictions that make them less desirable for private repositories. Google Code doesn't allow them at all, and GitHub currently charges a small but significant amount per private repository. Bitbucket charges by developer instead, with the first five being free. Another important consideration is that all three are backed by large commercial companies that are unlikely to blink out of existence overnight.

1. Preparation
OK, enough of the justifications, lets get migrating! In the Subversion world, the convention is that you structure your projects as follows:

foobar
   - Trunk
   - Branches
      - Europe
      - US
   - Tags
      - Version 3.11
      - Version 3.12

If you use different conventions (like "trunk" instead of "Trunk"), take extra care to translate these in the code below. Our example Subversion project URL is svn://myserver/myrepo/foobar so you will need to translate this too.

Next, we need to fix a small discontinuity in user handling between Subversion and Git. Subversion users are simple usernames (like gregf), whereas Git requires a user's name and their email address. Create a file called authors.txt containing an entry for each of your subversion users (usually listed in svn/myrepo/conf/passwd):

lisas = Lisa Simpson <lisa@duff.com>
barts = Bart Simpson <bart@duff.com>

Finally. make sure you have git and git-svn installed. Google to find out how to do this on your operating system.

2. Convert the Subversion Project to a Local Git Repository
At the command line, enter:

git svn clone svn://myserver/myrepo/foobar --no-metadata -A authors.txt -b Branches -T Trunk -t Tags foobar_local

Make yourself some tea, coffee or martini - this will take a wee while, as it works through the entire history of changes in the Subversion project. Assuming all goes well, you should end up with a directory called foobar_local containing both your project, and more importantly a hidden directory called .git. To check, change into the directory and view the log:

cd foobar_local
git log
git branch

That last command doesn't show your branches! Don't panic, this will come right later.

3. Convert Ignore Filemasks
If you have set up filemasks for Subversion to ignore, we can easily convert these into the Git equivalent:

cd foobar_local
git svn show-ignore -i trunk > .gitignore
git add .gitignore
git commit -m 'Convert svn:ignore properties to .gitignore.'

Note that Trunk in Subversion has been lower-cased by Git to trunk.

4. Create a Bare Git Repository
So far you have a local git repository, but you need a bare repository to clone from. As part of this process, we set Git's symbolic HEAD to what was Subversion's (lower-cased) trunk, tell Git about any branches, and rename trunk to master (the convention in Git).

cd ..
git init --bare foobar_bare
cd foobar_bare
git symbolic-ref HEAD refs/heads/trunk
cd ../foobar_local
git remote add bare ../foobar_bare
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare
cd ../foobar_bare
git branch -m trunk master

After the push, Git will list the new branches. I told you not to panic!

5. Convert Tags
Actually, if you use tags you might still be worried that all your tags seem to be branches. Git handles tags very differently from Subversion - in Git tags are simply aliases to a particular commit, whilst Subversion treats tags as a dead-end branch. Git-svn simply renames these branches, so we still need to convert these branch-tags to real Git tags:

cd foobar_bare
git for-each-ref --format='%(refname)' refs/heads/tags |
cut -d / -f 4 |
while read ref
do
   git tag "$ref" "refs/heads/tags/$ref";
   git branch -D "tags/$ref";
done

6. Push the Bare Repository to Bitbucket
Don't worry, we're nearly there now. The last step is to push all the branches up to Bitbucket. Log in to Bitbucket, and create a new repository called foobar. It will then show you the command required to clone this repository, which will include the HTTPS URL. The command will be something like this:

$ git clone https://myname@bitbucket.org/myname/foobar.git

The URL is the yellow section above. Now on your local machine, set the orgin and push the master (old trunk) and each branch to Bitbucket:

git remote add origin https://myname@bitbucket.org/myname/foobar.git
git push origin master
git push origin europe
git push origin us

Your project is now on Bitbucket.

7. Tidy Up
The only thing left now is to tidy up the local and bare repositories left on your machine.

cd ..
rm -rf foobar_local
rm -rf foobar_bare

And... you're done. Time for another tea, coffee, or martini to celebrate your success!

If this article has helped you, you can show your appreciation by telling any schools you are involved with about www.SchoolConferences.com. This simple online app has already revolutionised parent-teacher conference evenings for over a thousand schools. See how easy it is to book conferences with the demonstration event code HIGH4 for high schools, and ELEM4 for elementary schools. Thanks!

High-Replication Migration Lessons

2011-06-22T19:33:00.000-07:00

It's about a month since the migration, so how's it gone, what did we learn? What's it done to latency, error rates and CPU consumption? Most of all, was it worth the pain?

Lesson 1: Understand how eventual consistency will affect your application.

Our biggest fear was that there would be some inconsistencies between master-slave (MS) and high replication (HR) that would cause problems with our apps. And that fear was borne out when users started reporting that new objects were not showing up after they had created them. Some debugging found that the new objects did exist, and that we had been bitten by HR's eventual-consistency.

The datastore is much more efficient getting objects by ID than via a query, so I tend to give an object an array of IDs referencing child objects. When I added a new child, I ran a query to rebuild this ID array, which guaranteed it's consistency. With HR, the new child didn't show up in the query results for some seconds, so because I was rebuilding the array immediately after adding the new child, it disappeared. And when another new child was added, the previous one appeared - but the new one didn't. Users hate that sort of thing!

It was easily fixed - get_by_id() works as soon as an object is added, but queries do not. So we dispensed with the query, and simply added the ID to the array when we added a child (and removed it on deletion).

Lesson 2: Writes take longer and reads take about the same - but both are more consistent.

Probably the next biggest issue is performance. Writes take longer, probably about 50% on average, and reads are just about even. However, performance is far more consistent - our milliseconds/request graphs are much flatter.

With the MS datastore, we got occasional operations that took much longer than normal, and these seem to have largely disappeared with the HR.

Lesson 3: An order of magnitude less datastore timeouts.

The biggest difference has been on our error rate. With MS, we got a daily crop of datastore timeouts, and with HR they are extremely rare. With a non-relational database datastore errors lead to inconsistent data, because one entity is updated but the next may not. Users hate inconsistencies, so reducing errors by an order of magnitude is a huge win.

Lesson 4: HR costs more in CPU time.

Ah, but what about the cost? Google have equalised the storage costs between MS and HR, but you also pay for the CPU time to store and access your data. We're seeing a 30% increase in CPU after migrating, so HR will cost you more. How much depends on your application, but this may be also be moot when Google roll out their new instance-based billing.

So was it worth the time and cost involved in migrating from MS to HR? Yes. The reduced error rate has made our service far more robust, and reduced support calls. Also, we no longer need to manage user's expectations around Google's planned maintenance periods, which reduces a lot of customer irritation. These advantages easily outweigh the extra performance and CPU costs.

Finally, please leave a comment below about how your migration went, or if you have any questions about the process. If you'd like dedicated support with your migration, please contact me (greg at vig dot co dot nz).

If this article has helped you, you can show your appreciation by telling any schools you are involved with about www.SchoolConferences.com. This simple online app (built on Appengine of course!) has already revolutionised parent-teacher conference evenings for over a thousand schools. See how easy it is to book conferences with the demonstration event code HIGH4 for high schools, and ELEM4 for elementary schools. Thanks!

Migrating an App to High-Replication Datastore on Appengine

2011-04-26T19:20:00.000-07:00

Google's Appengine now gives you the choice between the original master-slave datastore, and the new high-replication datastore. In a nutshell, high-replication provides more reliable storage (in terms of data security and latency) but has slower write times and costs considerably more.

The owners of almost all serious applications will choose the high-replication datastore. The extra cost is a small price to pay (literally) to escape the maintenance outages, datastore timeouts and high-latency requests that plague the master-slave datastore.

The following article explains how to switch a production application from master-slave to high-replication with the minimum impact on your users.

Lets assume the app is called myapp, and it is currently available at the URL www.mydomain.com through Google Apps for your Domain. If you use myapp.appspot.com instead, you become reliant on Google to create an alias from the old app to the new one, and so have less control over the timing and length of the cut-over when your app isn't available to users. I'm also assuming you have billing enabled, because transferring data is likely to exceed the free quotas (particularly CPU time).

Many of the steps below involve switching between different Google accounts, in Appengine and Google Apps. Ideally you would log in to each account/service in a different tab before you started, but you can only have one account open in each browser. This means you either have to log in and out several times, or you can try running different browsers (Firefox, Chrome, IE) or different virtual machines for each account you need. In particular, the Datastore Admin page in Appengine admin may have a blank content section - signing out and in again usually cures this.

The biggest challenge in the migration is transferring all the data from myapp to myapp-hr. You probably don't want to create new entities with the high-level datastore API on myapp-hr because they will have different keys, and if you use keys or ids to link entities you'll need to map between the myapp keys and the myapp-hr keys. There are several possible approaches to this:

Use the appcfg.py utility to download data from myapp, and then upload it again to myapp-hr. This preserves keys, but is very slow because it is a single thread and all data has to be transferred across the internet to your machine and then back again - a two-stage serial transfer across many networks.
Write your own transfer utility with the Remote API so you can select which data to move at which time. This might allow you to limit the cut-over data transfer to active data only, and you can transfer the rest at your leisure during the preparation stage. However, this entails some fairly complex development and testing - you need to preserve keys (or map between old and new) and ideally have several threads running.
Write your own transfer utility using the Urlfetch API on myapp calling a handler on myapp-hr. As well as limiting the cut-over data transfer to active data only, it happens across Google's local network and both read and write occur simultaneously. This will be the fastest option, but again requires substantial development and testing.
Use the little-known Datastore Admin tool. This has all the advantages - across Google's local network, multiple threads and simultaneous read/write - except it transfers all data (you can select by entity kind, but usually each kind will have active and non-active entities so this isn't useful).
[UPDATE] There is now a built-in migration tool in the admin console. I haven't used this, but this would now be my first choice.

My attempts to use appcfg.py to do the transfer ended when it took over an hour to transfer a fairly small dataset. I then looked at developing my own transfer process, before a kind soul on the Appengine group pointed me to the datastore admin documentation. This transferred 140,000 entities (110MB) in 20 minutes - and chewed through 5 hours of CPU time!

If you have massive amounts of data, you may choose option 3 to minimise the cut-over transfer time (and the cost of the associated CPU time), but otherwise I recommend the datastore admin tool - it's simple, fast and safe. [UPDATE]The built-in migration tool in the admin console would now be my first choice.

OK, let's get started! Each instruction is colour-coded for Appengine admin, Google Apps admin, Google Apps mail, and shell/terminal.

The migration process is broken into three stages:

Preparation, which can be carried out at any time before the cut-over.
Cut-over, when your app is read-only for users.
Tidying up, done after the cut-over.

Preparation

First of all, we set up the new application, enable the datastore admin tool on the old and new apps, and test the data transfer process.

Create a new app called myapp-hr. Edit the storage options and select High Replication.
If your app sends emails, invite the sending address(es) to become developers for myapp-hr. Accept the invitation(s) in Google Apps for mydomain.com.
Enable billing for myapp-hr, and allocate resources as required. A large allocation for CPU time is a good idea - the transfer may require more than you expect.
Create myapp-hr source directory, and copy everything from myapp source directory.
Edit the new myapp-hr app.yaml:

Change application: from myapp to myapp-hr
Add builtins section:
  builtins:
  - remote_api: on
  - datastore_admin: on

Create file appengine_config.py containing (as a single line):
remoteapi_CUSTOM_ENVIRONMENT_AUTHENTICATION
('HTTP_X_APPENGINE_INBOUND_APPID',['myapp'])
Upload the new myapp-hr application:
appcfg.py upload myapp-hr
Enable datastore admin in myapp. Edit the original myapp app.yaml to add builtins section:
builtins:
- remote_api: on
Upload the myapp application:
appcfg.py upload myapp
Test the application is available by going to http://myapp-hr.appspot.com.
Test the data transfer process to find out how much real and CPU time it takes.Switch to myapp (not myapp-hr) application.
On the Datastore Admin page, select all entity kinds, and click "Copy to Other App".
Change {TARGET_APPID} to myapp-hr in target URL, leave extra header as is.
Note start time, go to the Task Queues page, refresh until default task queue is empty, and note end time.
Switch to myapp-hr application, and go to the Dashboard page to see how much CPU time the data transfer required.
Still in myapp-hr (double-check it's not myapp!), go to the Datastore Admin page, select all entity kinds, and click "Delete Entities" to clear all the data you just transferred.
Change your local copy of myapp so it shows a maintenance message.

Cut-Over

Now you know how long the data transfer takes, you can warn your users when and for how long the system will be under maintenance. I usually announce a pessimistic figure of at least twice what I think it will take, to cover any problems that might arise. If data transfer takes 10 minutes, you should be able to complete the cut-over in 15 - but I told users maintenance would take an hour. If everything goes well, users will be happy the outage was shorter than announced. Otherwise you have more time to resolve the issue or revert to the old application.

Here we go!

Upload the myapp application to show the maintenance message:
appcfg.py upload myapp
Switch to myapp application.
Set myapp to read-only mode by clicking "Disable writes..." on the Application Settings page.
On the Datastore Admin page, select all entity kinds, and click "Copy to Other App".
Change {TARGET_APPID} to myapp-hr in target URL, leave extra header as is.
Go to Task Queues page and wait until default task queue is empty.
Log in as admin, click on myapp in the Google App Engine section, and then delete www.mydomain.com under Web address.
Switch to myapp-hr application, go to the Application Settings page and click "Add Domain".
Accept agreement, click "Activate this service", add new URL "www" and click "I've completed these steps".
Go to www.mydomain.com and wait for the maintenance message to disappear, which means the domain is now pointing to myapp-hr. This should happen within a minute.
Test the application. On one occasion I found issues that disappeared in a few minutes (before I could properly diagnose them), so don't panic if odd things happen at first.

Tidy Up

All that's left to do is remove the datastore admin capability from your application, clear all the data from the old application, and remove it so it doesn't count against your application limit.

Delete file appengine_config.py from myapp-hr.
Remove datastore_admin and remote_api from builtins section of myapp-hr app.yaml.
Change the CPU time allocation to it's normal level on the myapp-hr Billing Settings page.
Switch to myapp application, and disable billing on the Billing Settings page. Also cancel recurring charge authorization. This will take several days to complete.
Go to the Application Settings page, and disable the application. You can then request permanent deletion, which will take 72 hours to occur.

Read our migration post-mortem.

Finally, please leave a comment below about how your migration went, or if you have any questions about the process. If you'd like dedicated support with your migration, please contact me (greg at vig dot co dot nz).

Installing Multiple Versions of Python on Linux

2011-04-02T20:26:00.000-07:00

The Google Appengine SDK requires at least version 2.5 of python, or else it spits the dummy. To compound matters, Google has announced that version 1.4.3 of the SDK is the last to support python 2.5, and all newer versions will require python 2.6. [Update: and now you need python 2.7 for multi-threaded apps.]

So how do you install later versions of python without impacting the system default version - which is required for all sorts of vital system services?

The trick is to use the prefix and altinstall options when building the new version of python. Make sure you have libsqlite3-dev and libssl-dev installed or Sqlite3 and SSL support won't be compiled in.

cd ~/src
wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2
tar -jxvf Python-2.7.2.tar.bz2 
rm Python-2.7.2.tar.bz2
cd Python-2.7.2/
./configure --prefix=/opt/python2.7
make
sudo make altinstall
sudo ln -s /opt/python2.7/bin/python2.7 /usr/bin/python2.7

You (and the system) will continue to use the old version by calling python, and you can use the new version by calling python2.7.

So to install sqlite3 support for python2.7 for instance:

cd ~/src 
wget http://pysqlite.googlecode.com/files/pysqlite-2.6.3.tar.gz
tar -xzf pysqlite-2.6.3.tar.gz 
rm pysqlite-2.6.3.tar.gz
cd pysqlite-2.6.3/
sudo python2.7 setup.py build_static install

Cascade Saddle 2010

2011-02-12T01:19:00.000-08:00

Click map to go to photo gallery

The Cascade Saddle track has a reputation of being one of New Zealand's harder walks, compensated with awe-inspiring views over the Matukituki valley, Mount Aspiring and the Dart Glacier. Who could resist?

My 16-year old sons Olly and Jasper are Venture Scouts, and part of their Chief Scout Award involves planning and carrying out a journey. They organised to take three days off their holiday jobs, but as the time came closer the weather forecast narrowed down our options. Luckily the Cascade Saddle was a possibility - I had wanted to do this walk for years, and the boys eventually succumbed to my arguments.

So as soon as Olly got off work on Wednesday 26th January, we loaded up the truck with our gear and set off - in pouring rain. It's about two hours drive from our home in Arrowtown to Raspberry Creek, and Nicole ensured our undying love by dropping us off - then driving all the way back again. After having my shorts licked by a cow as I did up my boots (a good omen?), we set off in showers to walk the two easy hours up to Aspiring hut. Arriving just before dark, we made some hot chocolate in this prince of mountain huts, chatted to a group of Dunedin Sea Scouts, and bunked down ready for the big day.

Day One
We were woken at 6:30 by raucous keas, only to find they had taken to Jasper's orthotic insoles. Luckily they were still mostly intact, and we were glad to see the weather had cleared according to plan. After a quick breakfast we set off at about 7:30.

The day was broken into four main sections: a steep climb from Aspiring Hut to the pylon; across Cascade Creek to the viewpoint; descending alongside the Dart Glacier to the valley floor; and a flat walk along the valley to Dart Hut.

The first section was a killer. The track was very steep, with frequent large steps that soon had my legs screaming. The boys... not so much. It became obvious that they were far fitter than I, stopping every few minutes to let their decrepit father catch up. We broke out of the bush after about two hours, getting stunning views of the Matukituki Valley and Mount Aspiring. Unfortunately the track got even steeper, and the tales of injuries and deaths on this section seemed only too credible. Steep bluffs were everywhere, and a stumble could easily be disastrous.

The pylon

The boys scampered on like mountain goats, and I toiled after them. The GPS was useful, mainly for the altitude readout - we paused for a few minutes after every hundred metres we gained. Eventually we crested the last bluff, the track mercifully levelled off, and we reached the pylon. At just over 1800m, we'd climbed 1400m in just over four hours. We munched some scroggin, and took in the incredible views - Raspberry creek was still in view, far below us, as was Aspiring Hut. Across the valley we could just make out French Ridge Hut, and we started plotting a visit. Above the hut stood Mount Aspiring cloaked in glaciers.

We then started to walk down a far more reasonable gradient to Cascade Creek. This runs through a delightful valley with snow grass meadows - maybe it was the euphoria of reaching the top, but it seemed to me to be absolutely idyllic. We reached the viewpoint after an hour, and settled in a small sheltered spot to wolf our lunch - fruit loaf with camembert! We took an hour to enjoy views from the precipitous edge, and laze in the sun under the watchful eye of a Kea.

After recharging our batteries, we set off down to the Dart Valley. Jasper's toes started to get painful in his cheap boots, but truly magnificent views of the Dart Glacier helped distract us. "Awesome" is overused these days, but that's what the glacier was. From the unblemished snow around the peaks, it ran down until the ice started cracking across the glacier as it rode over a lip of rock far beneath. Then longitudinal crevices appear as it is forced to turn a right angle by the Dart Valley, all the while picking up more and more shattered rocks from the valley sides. The tail is so encrusted with debris that it doesn't seem to be ice at all. We reached the tail after about an hour's walk down fairly loose rock on the south side of the valley.

As often happens, the last section was the hardest, despite being pretty much flat. We had put on sun-screen when we left the bushline that morning, but because we hadn't brought enough, we didn't apply more. The sun was hot, there was no breeze and the raw rock we were walking over reflected the heat straight back at us. Jasper's toes were very sore by this stage, so none of us really enjoyed the experience at this stage.

Which was a shame - it was almost like walking through time. At the glacier tail, no life existed and the valley was bare raw rock. But as we travelled down, mosses and small spruts of grass started appearing; then grass became thicker, and bush started appearing; and when we reached the hut 2.5 hours later, we were in mature beech forest.

We were all relieved - Jasper's boots came off in a flash, and we stumbled down to the river for a wash. The cold water was heavenly on our necks, arms and legs, and we realised how burnt we all were. All up we'd taken ten hours and ten minutes to get from Aspiring Hut to Dart Hut, including a good three-quarters of an hour idling at the viewpoint.

That evening was a bit of a blur - we rustled up some macaroni cheese with sliced cabanossi sticks, wolfed a pottle of mango on top of creamed rice, and hit the sack.

Day Two
After a solid night's sleep, we cooked porridge on the trusty coke-can cooker, and packed ready for the rain that was forecast. It was overcast as we set off, and after a hour it started drizzling. We marched on, through alternating beech forest and grassy flats, and the rain got stronger. My gaiters had kept my boots dry through all the streams the day before, but I stupidly didn't put them on when we started out. My boots got soaked just from brushing past the grass on the flats, and putting them on halfway along closed the stable door after the horse had bolted.

We didn't feel like stopping in the rain for lunch and so pushed on to Daley's Flat, arriving about 1:30pm. Despite the rain, the warden was busy with earthworks remodelling the path around the hut. It was a mud bath, and the voracious sandflies made it even less appealing to spend any time outside.

We had a leisurely lunch of soup, ryveta, cream cheese and sun-dried tomatoes. Then an afternoon snooze as the rain poured down, not waking till late afternoon. Dinner was dehydrated something (who can tell!) and a bar of chocolate. Jasper and Olly played werewolves (a version of mafia) with the rest of the hut inhabitants, then we went off for another good night's sleep.

Day Three

We woke to find the rain had stopped, and after porridge packed for the last time and rushed out the door to avoid the swarms of sandflies in the porch. The lunch we knew Nicole was bringing in was a constant driver keeping us moving, and we covered the ground at a good pace. Although it had stopped raining, the low cloud obscured Mount Earnslaw to our left. The track was similar to the previous day, with bush sections around bluffs followed by grassy flats.

And before we know it we were skirting Chinaman's Bluff, seeing parked cars through the trees, and sure enough there was Nicole waiting patiently for us. It was lovely to see her, and even lovelier to see the fine feed she had brought with her - bacon and egg pie and a potato salad!

It was a good trip. Apart from Jasper's toes and sunburn all round, we had no injuries, and my knees handled the climb and descent without a murmur. We got the weather right, doing the potentially dangerous Cascade Saddle on the one fine day we got. We learned not to leave orthotic insoles anywhere keas can get at them, and apart from Olly's mug cracking, our gear performed well. The boys did a sterling job of organising and executing the walk, and I was proud to have such capable companions for this unforgettable adventure.

DKIM for Google Apps (but not Appengine)

2011-01-15T20:07:00.000-08:00

In the ongoing war against spam, DKIM (Domain Key Identified Mail) seems to be the best candidate so far - if all legitimate email users started it, it would be the end of anonymous spam. This article explains DKIM, why you should implement it now, and how to do this with Google Apps.

DKIM allows receiving mail servers to reliably check an email is actually from the domain it says it's from, using a cryptographic key from your domain name service (DNS). This doesn't stop spam, but it makes senders responsible. If you get an unwanted DKIM signed message, you know exactly who sent it so you can take measures against them - blocking their domain perhaps, or suing them.

More and more spam filters are checking for DKIM signatures, and if your messages are signed they usually bypass other spam checks and are delivered immediately. So DKIM will improve your delivery statistics right now. And as more and more organisations implement DKIM, unsigned messages will become increasingly likely to be classed as spam, and so more and more organisations will implement DKIM to ensure their messages get through.

The faster DKIM spreads, the sooner anonymous spam will become history. We'll still need to manage spam because legitimate organisations will continue to do it, but after a few go bust from CAN SPAM fines even this should dry up. Bottom line - unless you are a spammer, it is very much in your interests to implement DKIM as soon as possible.

If you use Google Apps for your email, the good news is that DKIM is very easy to set up. Here's how...

Log in as the administrator to your Google Apps dashboard.
Click on the Advanced tools tab and then the link to Set up email authentication (DKIM).
Click on Generate new record.
Change the prefix selector from google to your organisation (if you want to - this isn't important at all).
This should result in a DNS Host name something._domainkey and a TXT record value v=DKIM1; k=rsa; t=y; p=lotsOfWeirdCharacters
Do not click the Start Authentication button yet!
Now go to your registrar's DNS control panel. I can't give exact instructions here because every control panel is different, but you want to find where you can set the A, CNAME, MX and TXT records for your domain name.
Add a TXT record with the host name and TXT record values from step 5. (Some control panels don't provide a
Wait a day or so to let the new DNS records percolate through the global name servers. If you're impatient and have dig, try dig txt something._domainkey.your.domain.name until it returns the key.
Log back in to your Google Apps dashboard, and go to the Advanced tools tab and the link to Set up email authentication (DKIM), and click the Start Authentication button.

Once you've done this, messages from your Google Apps domain* should contain a DKIM signature header. To test this, log in to your Google Apps account and send an email to another Gmail account. When it arrives, click the show details link - if all is well, there will be a new line saying Signed by giving your domain name.

* But not messages sent from Appengine, sadly. Please star this issue if this affects you. I'll update this post if the situation changes.

Start Here

2011-01-10T16:44:00.000-08:00

Crowded though the blogosphere is, I occasionally feel the need to make odd thoughts more public. By using a blog, I'm hoping you will add your own insights to what I hope will become a series of virtual discussions, rather than a list of my rants.

"Neogregious"? All the plays on "egregious" I could think of were taken, so this totally meaningless word will have to do. The background is the slopes of Mount Earnslaw near the Esquilant Bivvy at 2000m, taken on an epic trip with my brother Jon in 2005.

Samoa has more palm trees than Arrowtown

A little background about myself. I'm lucky enough to live in one of the most beautiful places in the world (Arrowtown in the heart of the Southern Alps of New Zealand), with the most gorgeous woman in the world (Nicole, a primary school teacher). We have two sons Olly and Jasper, who seem to be turning into healthy, pleasant and intelligent young men. I play the ukulele very badly, love tramping, and help run the Arrowtown Scouts.

After knocking off a Physics B.Sc. and an Electronics M.Sc. in the United Kingdom, I got my most important education when I took off round the world for two years. I worked on a bulk carrier from Greece to India, enjoyed the Buddhists in Dharamsala, developed noise calculation software in Singapore, sprayed for cockroaches in Broome, fixed Commodore 64s in Darwin, organised hundreds of cables for a SCADA system in Sydney, played catch-me-if-you-can with police launches around nuclear-powered warships in Sydney Harbour, and discovered that New Zealand was where I wanted to spend the rest of my life.

I also found that software development suited me, and luckily I suited software development too. Starting with C on DOS , working through C++ on Windows, back to C on Linux, PHP on LAMP and now... Python on everything!

I started my own company (Virtual Industries Group) in 2001 to exploit the shift from PC-based to online applications. One of our applications was a system that allowed schools to provide online bookings for parent-teacher interviews (or conferences in the US), and this really took off - we're doubling our customer base each year, and currently provide our service to schools in 8 countries around the world.

So welcome to Neogregious - I hope I can add more signal than noise, and that you will take the time to respond to posts with your own perspective.