Monitoring ESXi CPU/Memory via SNMP

I didn't find a particularly good resource for configuring and monitoring ESXi via SNMP, so here's what I did to get it to work. These instructions assume you want to use SNMPv3. You'll also need a host with vmcli installed on it.

1. create a local user on the esxi server by connecting directly to the esxi host using vsphere and going to the local users and groups tab. Right click somewhere in the user list and hit add
login: cacti (my poller)
password: somepassword

2. From the vCLI host, issue the following commands
esxcli -s esxiserverhostnameorip -u root -p rootpassword system snmp set --authentication [none,SHA1,MD5]
optional - to configure privacy do
esxcli -s esxiserverhostnameorip -u root -p rootpassword system snmp set --privacy [none,AES128]

3. Configure users by issuing the following commands
esxcli -s esxiserverhostnameorip -u root -p rootpassword system snmp hash --raw-secret --auth-hash yourdesiredsnmpv3authpass
optionally add --priv-hash yourprivsecret if using privacy
Hashes will be generated and displayed on screen, then do:
esxcli -s esxiserverhostnameorip -u root -p rootpassword system snmp set --users userid/authhash/privhash/security
where userid is the user in step 1, authhash is the authentication hash, privhash is the privacy hash (or use a single dash "-" if not using privacy) and security is either none, auth, or priv

4. Test the user by executing the following:
esxcli -s esxiserverhostnameorip -u root -p rootpassword system snmp test --user username --raw-secret --auth-hash desiredsnmpv3authpass (also include --priv-hash if you're using privacy)

5. enable SNMP by executing:
esxcli -s esxiserverhostnameorip -u root -p rootpassword system snmp set --enable yes

Now you can monitor your ESXi host via SNMP, some useful OIDs

1.3.6.1.2.1.25.2 - hrStorage
Not only contains information about datastores, but one of the entries is "Real Memory" which contains information about memory usage.

1.3.6.1.2.1.25.3.3 - hrProcessorTable
Contains CPU usage values for each core on the system

.1.3.6.1.2.1.2 - Interfaces MIB
Information on various network interfaces defined on the host

VMware documentation on configuring SNMP: https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.monitoring.doc%2FGUID-2E4B0F2A-11D8-4649-AC6C-99F89CE93026.html

Modifying Bell Paging Schedules in Asterisk

The site that caused me to set up bell paging on our trixbox system also threw me a curveball by having an "alternate" schedule that they needed implemented every Thursday, although occasionally they need to set the "Thursday" schedule on other days of the week. I got tired of manually updating the crontab for these times, so I configured a feature code so that they could override the normally programmed time themselves.

First create a custom context to use as a destination that will run the script that modifies our crontab
edit /etc/asterisk/extensions_custom.conf ad add something like
[ext-belltoggle-custom]
exten => s,1,AGI(toggle-bells.sh)

Next create a custom destination to trigger that context
Custom destination : ext-belltoggle-custom,s,1
Description: Toggle Bells

Then create a Misc Application to trigger our custom destination:
Description: Toggle Bells
Feature Code: *281 (or whatever you like)
Feature Status: Enabled
Destination: Custom Destinations -> Toggle Bells

Create toggle-bells.sh in /var/lib/asterisk/agi-bin with the following

#!/bin/bash

# Consume all variables sent by Asterisk
while read VAR && [ -n ${VAR} ] ; do : ; done

# Answer the call.
echo "ANSWER"
read RESPONSE

# Say the letters of "Hello World"
#echo 'SAY ALPHA "Hello World" ""'
#read RESPONSE

if crontab -l | grep -q "ALTERNATE"; then
  echo "51 * * * * /var/lib/asterisk/bin/freepbx-cron-scheduler.php
1 00 * * 1,2,3,5 /usr/bin/crontab /var/lib/asterisk/scripts/normalbells
1 00 * * 4 /usr/bin/crontab /var/lib/asterisk/scripts/alternatebells
1 00 * * 0,6 /usr/bin/crontab /var/lib/asterisk/scripts/weekendbells
##NORMAL BELLS##
##Put your normal schedule here with lines like the following##
30 08 * * * /usr/sbin/asterisk -rx 'originate Local/8102@ext-paging extension 8103@ext-paging'
" | crontab -

  echo 'EXEC PLAYBACK "custom/NormalBellActivated" ""'
  read RESPONSE

elif crontab -l | grep -q "NORMAL"; then
  echo "51 * * * * /var/lib/asterisk/bin/freepbx-cron-scheduler.php
1 00 * * 1,2,3,5 /usr/bin/crontab /var/lib/asterisk/scripts/normalbells
1 00 * * 4 /usr/bin/crontab /var/lib/asterisk/scripts/alternatebells
1 00 * * 0,6 /usr/bin/crontab /var/lib/asterisk/scripts/weekendbells
##ALTERNATE BELLS##
##Put your alternate schedule here with lines like the following##
35 08 * * * /usr/sbin/asterisk -rx 'originate Local/8102@ext-paging extension 8103@ext-paging'
" | crontab -

  echo 'EXEC PLAYBACK "custom/AlternateBellActivated" ""'
  read RESPONSE

fi

exit 0

Create 3 crontab files in /var/lib/asterisk/scripts/ called normalbells alternatebells and weekendbells that will be sourced to set the schedule to the normal time each day.

They should look identical to what the asterisk user's crontab looks like when it's in one of the modes.

Finally, create two system recordings named NormalBellActivated and AlternateBellActivated that will be used to inform the user which mode they're in.

FreePBX BLF Time Conditions

My previous post was about increasing the number of Call Routing toggles in FreePBX because I thought that was what I would need to do in order to get BLF (Busy Lamp Field) working with the presence feature on our Polycom SoundPoint phones. The reason I wanted to do that was that so users could have an indication if their phone was in "night mode" or not. As it turns out there is a MUCH better way to handle this.

Step 1
In FreePBX, go to Settings -> Advanced Settings and change "Enable Custom Device States" to True

Step 2
In Applications -> Time Conditions, create your Time Condition. You'll notice that by enabling the previous option, you now have a "Generate BLF Hint" checkbox. Be sure to set that to ON, make sure you also set the "Enable Override Code" box to ON. Once you save the Time Condition, make a note of the override code assigned to it (likely something like *270)

Step 3
On your phone, enable the presence feature. For my SoundPoint IP 601, this meant setting         feature.1.enabled="{$feature_1_enabled|1}" in my server.cfg
I had tried to set feature.1.enabled="1" in sip.cfg which I had seen referenced multiple times, however it never seemed to enable the presence feature.

Step 4
Add the extension *270 to the list of watched extensions on your phone. You can provision it via config file, or set up the extension in the local directory on the phone, and while doing so, scroll down to the "Buddy Watch" option and toggle it to "Enabled"

Increasing the number of FreePBX Call Flow Control Options

We're starting to retool our VOIP infrastructure, and consolidate down from 30+ Trixbox servers to a single FreePBX server. One issue with the amalgamation however is that by default, FreePBX only allows you 10 Call Flow Control indexes. Since we have some 30 odd sites, this wouldn't work for us. However, this fantastic blog post: http://sysadminman.net/blog/2013/daynight-aka-call-flow-control-more-than-10-4884 shows you how to increase that to the number of your choosing. I'll repost the gist here.

1. copy /var/www/html/admin/modules/daynight/functions.inc.php to /var/www/html/admin/modules/daynight/functions.inc.php.dist (good to have a backup)
2. edit /var/www/html/admin/modules/daynight/functions.inc.php and find function daynight_get_avail() { (should start around line 258)
3. under global $db; add another line that reads $NUMDAYNIGHT = 99;
4. comment out the section that starts for ($i=0;$i<=9;$i++) { and ends return $list; (since this is PHP, just add // to the start of those lines
5. under the commented out lines, add the following:

         for ($i=0;$i<=($NUMDAYNIGHT-1);$i++) {
                 $n=ceil(log10($NUMDAYNIGHT));
                 $format="%0".$n."d";
                 $j=sprintf($format,$i);
                 if (!in_array($j,$results)) {
                         $list[]=$j;
                 }
         }
         return $list;

The whole block of code from line 258 function daynight_get_avail() { to line 284 } should look like:

function daynight_get_avail() {
        global $db;
        $NUMDAYNIGHT = 99;

        $sql = "SELECT ext FROM daynight ORDER BY ext";
        $results = $db->getCol($sql);
        if(DB::IsError($results)) {
                $results = array();
        }

//      for ($i=0;$i<=9;$i++) {
//              if (!in_array($i,$results)) {
//                      $list[]=$i;
//              }
//      }
//      return $list;

        for ($i=0;$i<=($NUMDAYNIGHT-1);$i++) {
                $n=ceil(log10($NUMDAYNIGHT));
                $format="%0".$n."d";
                $j=sprintf($format,$i);
                if (!in_array($j,$results)) {
                        $list[]=$j;
                }
        }
        return $list;
}

Now if you go to Applications -> Call Flow Control you will have 99 options for Flow Toggles!

Scheduled Switch Reloading

Had this tip passed on to me by an Axia representative when we were provisioning a bunch of changes to our network. if you're making switch configuration changes remotely (something I often have to do because of how remote our sites are) a good idea is to schedule a reboot of the switch in the near future, then make your configuration changes. Then if everything looks good, write your config and cancel the reload. That way if things go bad and you loose connectivity, the switch will reboot with it's old (working) configuration!

For Procurve switches, this might look something like:
reload after 35 (to schedule a reboot in 35 minutes)
and if you want to cancel the reload:
no reload

Bells via Asterisk Paging

As a few of our sites don't have a traditional PA system, and no way to play bell noises to signal class breaks, we decided to use the asterisk paging system to play bell noises through the handsets in each classroom. There are a few different ways of doing this, but here's the solution we came up with

1) Create a paging group with the extensions you want to play the sound (our sites are currently using Trixbox, so this is under the Paging and Intercom section). Let's call that extension XXXX

2) Create a dummy extension to play the bell noise.
This involves editing /etc/asterisk/extensions_custom.conf and creating an extension that just plays the sound and then hangs up. Let's call that extension YYYY. Add the following near the end of the file:

[ext-paging-custom]

exten => YYYY,1,Answer

exten => YYYY,n,Wait(3)

exten => YYYY,n,Background(hello-world)

exten => YYYY,n,HangUp()

Replace hello-world with the name of a sound file in your asterisk system. If the sound takes too long to start, or has already started when the page happens, you'll need to adjust the value for Wait() as well.

3) Create a cron job to make the bells ring. You can use the following command
asterisk -rx 'originate Local/YYYY@ext-paging extension XXXX@ext-paging'
This will originate the page from the YYYY extension, which will play the bell noise and hang up.

It's also not a bad idea, if you're running trixbox or freepbx or similar, to add the YYYY extension to the "custom extensions" (trixbox: PBX -> PBX Settings -> Tools tab -> custom extensions, freepbx: Admin -> Custom extensions) so that you don't accidentally assign extension YYYY.

Syncing Active Directory Groups as Moodle Cohorts

We had a situation that required us to synchronize certain active directory groups with enrollment in a moodle course. The way we accomplished this was to use a moodle userscript that synced active directory groups as cohorts. Then we assigned the cohorts to the course using the "cohort-sync" enrollment type.

The following steps assume you already have moodle LDAP auth configured and are running the /auth/ldap/cli/sync_users.php to sync LDAP users to moodle.

1. Obtain the sync_cohorts.php script from https://tracker.moodle.org/browse/MDL-25011?attachmentOrder=desc and place it in /auth/ldap/cli/

2. Add the following lines to moodle's config.php

$CFG->ldap_group_class='group';
$CFG->ldap_process_nested_groups=1;
$CFG->ldap_real_user_attribute='sAMAccountName';

ldap_group_class - needs to be set to 'group' for active directory in order to match user groups
ldap_process_nested_groups - set this to 1 if you have groups as members of your groups in order to locate the actual users
ldap_real_user_attribute - this is probably whatever you have "User attribute" set to in moodle's LDAP settings. Also note that it is CASE SENSITIVE (that one gave me some trouble until I realized what the issue was).

3. Add the sync_cohorts.php scrip to your webserver user's crontab to be run sometime after the ldap sync_users.php finishes.

4. Add the newly created cohort to your course using the cohort-sync enrollment type.

Converting from Hyper-V using VMWare Converter: Take 2

I ran into another cause of the dreaded “Unable to obtain hardware information” when trying to convert some VMs from a Hyper-V 2012 server. Turns out that VMware Converter will throw this error if the VM is using the newer .vhdx virtual disk format instead of .vhd.

To get around it, convert the disks to .vhd (because Hyper-V server is a core server, I used the PowerShell cmdlet "Convert-VHD".

Power down the VM

Convert the vhdx to vhd using the command: Convert-VHD - Path C:\Path\To\Virtual\Drive\VMname.vhdx -DestinationPath C:\Path\To\Virtual\Drive\VMname.vhd

Reconfigure the VM. Detach the old vhdx drive, attach the vhd drive.

Run converter again.

Presto!

Converting from Hyper-V using VMWare Converter

Ran into this very issue that was blogged about here: http://www.techromeo.com/?p=115

Lots of head scratching on this one.  As usual, I took the long way around so I figured I had to share it.
Problem:  When using the new VMware Converter Standalone, running on Windows 2008R2, to convert Hyper-V machines, I would receive the error “Unable to obtain hardware information”.
Solution:  On the folder on the Hyper-V host that contains the vhd’s, set the NTFS permissions so that the local Users group has full control, rather than the default read permissions.
Background:  It appears that VMware converter creates a local account on the machine it is installed on.  I tried running Converter locally on the Hyper-V host and setting explicit permissions to that local account.  Also keep in mind that Converter will ask you for credential to connect to the Hyper-V host and will not work without them.  I also tried to set the permissions to the machine account to that of the machine running converter.

 Good to know!

Syncing Moodle Users from Active Directory

I've been working a lot lately on automating user creation in moodle. We already have LDAP auth against our Active Directory server, but that only creates users when they log in. Also, because we are using Mnet to handle logins to two other servers, accounts weren't getting pre-populated there.

I came across a great powershell script by David Wiseman that I could adapt to query our student information system to create accounts in active directory. You can check it out here: http://www.wisesoft.co.uk/scripts/powershell_create_ad_user_accounts_from_sql_server_database.aspx

Once I had my accounts being created, I could use moodle's sync_users.php script running as a cron job to pre-populate our main moodle server. Some caveats though. Make sure you modify your objectclass filter in moodle to be (&(objectClass=user)(!(objectClass=computer))) otherwise you'll also get computer accounts as well as user accounts. You'll also either need to patch sync_users.php to allow paginated results from LDAP or increase the MaxPagesize variable for your active directory install to be able to return all the users in a single page using ndsutil (the default is 1000 results per page). There's some relevant moodle discussion here.

The last bit was to be able to pre-populate the linked mnet servers with the proper accounts. Which I did using the following SQL query

INSERT INTO moodle_db2.mdl_user (auth, confirmed, policyagreed, mnethostid, username, password, idnumber, firstname, lastname, email, institution, city, country)
SELECT 'mnet', mdl_user.confirmed, mdl_user.policyagreed, '5', mdl_user.username, mdl_user.password, mdl_user.idnumber, mdl_user.firstname, mdl_user.lastname, mdl_user.email, mdl_user.institution, mdl_user.city, mdl_user.country
FROM moodle_db1.mdl_user
WHERE institution = 'Mnet Institution Name'
ON DUPLICATE KEY UPDATE auth='mnet', confirmed=moodle_db1.mdl_user.confirmed, policyagreed=moodle_db1.mdl_user.policyagreed, mnethostid='5'

This lets me select just the users who should be on the federated mnet server and create their accounts, if the user already exists, it updates a few fields. mnethostid will vary by installation, in my case it was 5 but you should be able to query the database easily to find out what it should be in your specific installation.

Large Scale Course, Enrollment & Content Management

Notes from Moodlemoot 2013

Asim Aziz and Chris Goetz from the U of A presented

Challenges
Info from SIS is very fluid
18 different faculties
SIS info is often behind reality
Philosophy is if something is not urgent and not unusual, it should be automated.

Automation
course creation - automatically created upon instructor request
self-service request tool

Content management
copy content from old courses, user never sees moodle's backup restore page

enrollment management
180000 seats in flux
cohorts used for enrollment
instructors/TAs enrolled
sometimes a faculty level course used for shared resources

create course
copy contents
add cohorts

faculties broken down by categories
cohorts sync'd daily
groups created with cohort

copy content plugin + IMS enterprise plugin

Archive server filesystem is mounted to production server to allow access to courses (so it can get at MBZ)

Automation Using SQL Server

Notes from Moodlemoot 2013

Jim Bennett from Greater Saskatoon Catholic School board presented

Uses MSSQL as all their existing servers were Microsoft, and expertise designing and maintaining asp apps
Server runs moodle 1.9 due to the amount of customization
Custom apps are in folders prefixed with division name
Custom blocks in moodle

courses run for 150 days in year (separate run of 2 months over summer)
students are enrolled for 150 days (needs to account for holidays/summer/etc)
every two weeks a report card is emailed to students with marks for activities
at 75 days midterm report card
at 150 days final report card is sent to teacher for approval before sending
"attendance" is tracked as more than just log onto the system uses moodle logs to see if they actually did anything
a 5 days misses message sent to student, parents, teacher
at 10 day missed same but more strongly worded
at 15 days warning goes to counselors and admin
Using maplewood for SIS
Stored procedures are generic and modulers
Self contianed accounts, no central auth

Why, When, How to Upgrade Moodle

Notes from Moodlemoot 2013

Olivier Guillot and Gilles-Philippe Leblanc from Université de Montréal presented

Why:
Bugfixes, security fixes
major improvements
stay close to current to make upgrades easier (ie 2.1 to 2.5 would be hard)
easier on any custom mods
easier to backport critical fixes from upcoming verisoin
avoid pitfall of being too far behind to upgrade

when:
avoid 2.x.0 versions
2.x.1 is usually out in a month with substantial fixes
each version has new improvements that are subject to bugs
january and august upgrade windows for major versions 2.x

how:
get neve version
merge custom code on new
commit sources
insall on devel server first, then road test

do functional testing using browsers
test critical modules (grades assignment forums)
fix/merge theme, fix functional test
backport blocker and critical issues from upcoming versojns
load testing to see if new ver has performance issues
winmerge
3 month testing phase

minor fixes - backoirt security blocker critical
search tracker for updates marked with UI change -> that way you know what to make users aware of
upgrade based on downtime from stats
post on form, post news block on all pages about upcoming downtime

Integrating Moodle With Facebook

Notes from Moodlemoot 2013

Rafael Scapin from Dawson College presented

Pilot project with a single course
They created a closed facebook group and added all the students to it.
The teacher creates a separate facebook account to manage the group with.
The teacher would post course updates to the facebook group so that students could get notifications.
Some issues:
Some students not comfortable with friending their teacher
Facebook changes all the time, sometime tools would disappear

The moodle server still holds all content, the facebook group just links into it.
The primary draw for it for students was being able to get notifications on their phone when new content was posted. Will this be a non-issue when the new moodle app is released?

There is a site called academicconnect.net which is a social network for educators and students.

Sustaining Moodle in a Large Organization

Notes from Moodlemooot 2013

Dave Sun from the U of A talked about their Moodle installation

The U of A migrated from Blackboard Vista to Moodle

They use three primary moodle servers, production, archive and test
archive is a copy of the expired course with all student data removed, used for staff to import course content from.

Git is used to maintain moodles.

The instances run as a vmware cluster, with pgsql as the database.

There is a provincial initiative to consolidate moodle hosting with Cybera. Cybera is using openstack and Chef to roll moodle installs.

In addition to the 6 application servers and single database server, there is a server dedicated to running moodle cron and backup jobs.

Redmine is used for project management.

Tuesday AM is set time for minor changes like plugin updates. Last sunday of month for major upgrades.
Moodle's cohort system is used for handling enrolments.

Users are kicked across in a daily batch
Using PeopleSoft on the back end
Help desk uses Kayaco support suite for ticketing with end users.
OTRS is also used for ticketing
Baseline response to tickets is 24h
Very few core modifications (maybe 10 lines of code total, and that's for their category level admin plugin)

0 to 100 in 18 Months

Notes from Moodlemoot 2013

Grzegorz Dostatni presented on the U of A's moodle infrastructure.

Highly available infrastructure
Organizational structure is as if not more important than the technology used. You need to strike a balance between developers "fix it now, put it in prod now" and sysadmins "must always remain stable".
Cyber is hosting the U of A, NAIT and Athabasca U moodle instances.
Using emc cx-4 for storage, U of A reviews storage vendors every 5-7 years.
6 physical hosts, 2 in a separate data center
vmware is set to never run app nodes on the same server to maintain redundancy
using ubuntu LTS for OS
pgsql used because of in house familiarity, caused some headaches as mysql is more widely supported
db is running in a vm, not a physical server
lvm fileserver replicatiion
drdb to mirror over network
hourly backups with 30 day retention using dedup on fileserer
eAccellerator for PHP acceleration
both servers are NTP sync'd as to restore to a certain point, the db and fileserver must be restored to the same point in time

Data points to monitor:
On all servers
cpu load
disk
free memory

On db server
sql errors
long processes (greater than 3h runtime)

On fileservers
drdb status/errors

On app servers
num apache processes (>100 = yellow alarm, >125 red alarm)

Have a process in place for dealing with bottlenecks (ie. first we upgrade ram, then cpu then disk)
Scripts: www.ualberta.ca/~dostatni/moodlemoot2013
NFS mounts to filestorage

43 000 Students – Which architecture?

Notes from Moodlemoot 2013

Olivier Guillot presented on Université de Montréal's moodle installation

At U of M, moodle is their 2nd most critical system after email.

The have a similar setup to U of A in that production instance is loadbalanced, and moodle cron is run on a separate server.

Using moodle's integrated backup system to do daily course backups, history of 10 backups

Using a PHP cache is necessary (doubles performance) U of M uses APC (alternate PHP cache)

Using mysql tuner/mysql enterprise monitor to monitor database health

Daily push of enrolments

courses are named with coursecode-section-semester (ie. MAT10101-A-W13)

currently in house SIS, moving to peoplesoft

single moodle instance

using munin for graphing, nagios for alterting, moving to git for version control/updates, using Hudson (now Jenkins) for server mirroring.

Moodle-Stats for monitoring instance (github moodle-scripts)

agile-scrum and jira/confluence for helpdesk

Leaves courses open for 3 years so that students can still audit them

Teachers responsible for importing content into new courses.