0 to 100 in 18 Months

Notes from Moodlemoot 2013

Grzegorz Dostatni presented on the U of A's moodle infrastructure.

Highly available infrastructure
Organizational structure is as if not more important than the technology used. You need to strike a balance between developers "fix it now, put it in prod now" and sysadmins "must always remain stable".
Cyber is hosting the U of A, NAIT and Athabasca U moodle instances.
Using emc cx-4 for storage, U of A reviews storage vendors every 5-7 years.
6 physical hosts, 2 in a separate data center
vmware is set to never run app nodes on the same server to maintain redundancy
using ubuntu LTS for OS
pgsql used because of in house familiarity, caused some headaches as mysql is more widely supported
db is running in a vm, not a physical server
lvm fileserver replicatiion
drdb to mirror over network
hourly backups with 30 day retention using dedup on fileserer
eAccellerator for PHP acceleration
both servers are NTP sync'd as to restore to a certain point, the db and fileserver must be restored to the same point in time

Data points to monitor:
On all servers
cpu load
disk
free memory

On db server
sql errors
long processes (greater than 3h runtime)

On fileservers
drdb status/errors

On app servers
num apache processes (>100 = yellow alarm, >125 red alarm)

Have a process in place for dealing with bottlenecks (ie. first we upgrade ram, then cpu then disk)
Scripts: www.ualberta.ca/~dostatni/moodlemoot2013
NFS mounts to filestorage