CS Cart git hook script to auto-flip

One of the biggest problems I’ve had in CS Cart development was remembering to disable and then re-enable any addons that had changes. I’ve created some scripts to help the devops development flow in CS Cart:

https://github.com/heybige/cscart_cli_scripts

‘cli_lib.php’ is a stripped down version of ‘admin.php’ that I use to bootstrap into CSC from the command line. ‘clear_cache.php’ should be self-explanatory – it just clears the cache from the command line.

HINT: if you’re having any kind of problem with CS Cart, always try clearing the cache. And you may consider doing a “hard reset” (rm -rf var/cache/* in your root CSC directory) because there are files that don’t get cleared out with a standard “clear cache” command.

So the magic starts with a git hook script ‘post-merge’ (.git/hooks/post-merge):

#!/bin/bash

export ROOT=/path/to/cscart/base

## HARD RESET
rm -rf $ROOT/var/cache/*

## RUN COMPOSER
cd $ROOT/app/lib
composer install

$ROOT/bin/flip_addon.sh

Pretty standard. It’s worth noting that you can add your own Composer packages in the existing composer.json file and use them throughout CSC. I’ll do a future post about how I added some custom logging to CS Cart (lack of file-based logging is another of my huge CSC pet peeves).

#!/bin/bash

export ROOT=/path/to/cscart

ADDONS=()

ADDON_LIST=`cd $ROOT; git diff "HEAD@{1}" --name-only | egrep 'app/addons/.+?/addon.xml'`

if [ ! -z "$ADDON_LIST" ]; then
    for file in $ADDON_LIST; do
        OIFS=$IFS
        IFS='/'
        arrItems=($file)
        ADDONS=("${ADDONS[@]}" ${arrItems[2]})
        IFS=$OIFS
    done
fi

ADDON_LIST=`cd $ROOT; git diff "HEAD@{1}" --name-only | grep 'var/themes_repository/basic/templates/addons/'`

if [ ! -z "$ADDON_LIST" ]; then
    for file in $ADDON_LIST; do
        OIFS=$IFS
        IFS='/'
        arrItems=($file)
        ADDONS=("${ADDONS[@]}" ${arrItems[5]})
        IFS=$OIFS
    done
fi

function join { local IFS="$1"; shift; echo "$*"; }

if [ ! -z "$ADDONS" ]; then

        if [ ${#ADDONS[@]} -gt 1 ]; then
                CHANGED=($(printf "%s\n" "${ADDONS[@]}" | sort -u))
                LIST=`join : "${CHANGED[@]}"`
        else
                LIST=${ADDONS[0]}
        fi

    echo "php $ROOT/bin/php_flip_addon.php $LIST"
        php $ROOT/bin/php_flip_addon.php $LIST
fi

Basically, this script just determines which addon files have changed, and builds a (unique) colon-separated list to pass to ‘php_flip_addon.php’

I won’t post ‘php_flip_addon.php’ but it takes the colon-separated addon list, starts building a stack of the addons that need to be “flipped”, taking into account any dependencies. It uninstalls all those addons in the proper order, then re-installs them in the reverse order. If there’s a problem uninstalling, it will immediately reverse the order to try and get back to the “known good” state.

Ansible trick for spinning up a new server

So I think everyone and their mother has fallen in love with Ansible – I know I have! Mostly because I’m not really a fan of Ruby and Ansible is just so simple and basic to operate – ssh only required.

I’ve got a bunch of Ansible roles defined: apache-php5, nginx-fpm, nginx-hhvm, etc. It’s nice to be able to spin up a server and test things out. Here’s a quick little script I use to execute roles against a server not listed in /etc/ansible/hosts file:

#!/bin/bash

if [ -z $2 ]; then
  echo "$0 [www.domain.com] [role]"
  exit 1
fi

ansible-playbook $2.yml -i "$1," --extra-vars "fqdn=$1"

The trick is the “$1,” which allows you to define the host on the fly instead of having to define it in the ‘hosts’ file.

All my roles use the ‘aws‘ module to spin up a new EC2 instance, create a new DNS A record, and configure the new host to the specified role as the referenced FQDN.

Setting up DNS on Route 53

I have always managed my own DNS on servers that I control because it was something I was capable of doing, and I hate being dependent on someone else when something needs to be done. But I’ve been periodically working on the load time for my blogs (this one and http://www.heybige.com), and the performance for my primary and backup DNS servers just isn’t cutting it any more.

My DNS server hosts are small, inexpensive, out-of-the-way hosting companies, and the servers are either older, slower servers or VPS – great on the pocketbook, but not great for performance. The hosts are far from the backbone, so it takes a number of hops just to get to major exchange points, all things that slow down DNS lookups.

The easiest solution to this problem is to host your primary domains’ DNS at Amazon’s Route 53.

I ran a series of tests from http://www.webpagetest.org and the best DNS performance I could muster was 122ms (and the average was around 350ms). Once I changed over to Route 53, the average DNS lookup was around 25ms, which make a significant difference in the page load time.

Note: this significant packet delay carries through to all aspects of the page loading. Each packet coming in and going out is subject to this same delay. The solution is to host your website on fast servers on or close to major NAPs, but for me, I’m happy hosting everything on a cheap server that I fully control.

Openssh 6.2 allows for both public key and password authentication

The concept of using public/private keys to bypass password entry requirements always sounds good in theory, but my security conscious would never allow me to do so, on the fear that someone who has access to one server can serially access the rest of your server installations.

I do use public key crypto for certain things, like having a separate Subversion user/key so I’m not prompted for a password when I’m committing code.

I always thought, why can’t we have both public key and password authentication on an account? I knew there were patches to make that happen, but who wants to deal with patches every time openssh is updated?

The latest version of openssh (6.2) has answered my prayers. You can enable the requirement that the public key be valid AND that the user authenticates with a password. Add the following line to your ‘sshd_config’ file:

AuthenticationMethods publickey,password publickey,keyboard-interactive

I highly encourage all sysadmin to enable this. I used to watch my system logs getting blasted on a daily basis from brute force guessing on my sshd daemon, but it comforts me greatly to know hackers aren’t even getting a chance to brute force passwords unless they have the proper public key:

error: Received disconnect from a.b.c.d: 11: Bye Bye [preauth] : 2460 time(s)
error: Received disconnect from e.f.g.h: 11: Bye Bye [preauth] : 1428 time(s)

Those requests were rejected before even getting a chance to authenticate. I still get prompted for a password from my main computer so there’s not an open link to my servers from this computer. Also, you can selectively enable ‘disable’ the password requirement for certain accounts. I added the following lines to ‘sshd_config’ as well:

Match User subversion_user
AuthenticationMethods publickey

This allows the ‘subversion_user’ (a limited access user) to authenticate ONLY with the public key and not be prompted for a password.

ban/unban scripts for ipf firewall

System administrators are always running across unauthorized access requests and reasons to ban/block IP addresses – on webservers, daemons, etc. I’ve long used some homemade scripts to facilitate this and figured I would include them here.

Here’s the ‘ban’ script, which just takes an IP address to ban via ipf (e.g. “ban 1.2.3.4”):

#!/bin/bash

CIDR=32
CONF=ipf.conf

IP=`echo $1 | /bin/tr -d '[:alpha:]\:[:space:]'`

ESC_IP=`echo $IP | /bin/sed 's/\./\\\./g'`

EXISTS=`/bin/grep "$ESC_IP" /etc/ipf/$CONF`

if [ -n "$EXISTS" ]; then
        echo "$IP is already blocked"
        exit
fi

REGEX="\.0$"

if [[ $IP =~ $REGEX ]]; then
    CIDR=24
fi

CMD="block in quick from $IP/$CIDR to any"

echo $CMD | /usr/sbin/ipf -f -

/bin/echo $CMD >>/etc/ipf/$CONF

And here’s an unban script which reverses the ban:

#!/bin/bash

CIDR=32
CONF=ipf.conf

IP=`echo $1 | /bin/tr -d '[:alpha:]\:[:space:]'`

ESC_IP=`echo $IP | /bin/sed 's/\./\\\./g'`

EXISTS=`/bin/grep "$ESC_IP" /etc/ipf/ipf.conf`

if [ -z "$EXISTS" ]; then
        echo "$IP is not blocked"
        exit
fi

REGEX="\.0$"

if [[ $IP =~ $REGEX ]]; then
    CIDR=24
fi

CMD="block in quick from $IP/$CIDR to any"

echo $CMD | /usr/sbin/ipf -r -f -

/bin/echo $CMD >>/etc/ipf/unban_log

perl -pi -e "s/block in quick from $ESC_IP\/$CIDR to any\n//" /etc/ipf/ipf.conf

These files (and the init.d startup script) are at https://github.com/heybige/ipf-ban-unban