Setting up Exim to Use Amazon Simple Email Service (SES)

Amazon Simple Email Service (SES) is a Email Marketer’s dream come true. It allows you to send bulk emails to customers without fear of having your email land in their spam/junk folder — unless of course you’re sending spam and a huge majority of your “customers” start marking it as such.

More to the point, it’s very easy to setup SES with sendmail and postfix. However, if you’re like most small/medium businesses, you’re using cPanel to host your site and that comes with Exim as the mail server. Of course, Amazon forgot to add instructions for Exim and so, here you are. Let’s begin. Read more…

Site Re-design

New Site Design

Last time I did a custom re-design for my site was way back during my blogspot time. That was in 2006 — five years have passed but I still like the design. When I moved to wordpress.com, I didn’t have a way of creating my own design so I stuck with the best design I could find. I moved to my own host here at CSRDU last year but didn’t really feel the need to create a custom design. The result, even with the great theming mechanism provided by WordPress, I never wrote a custom theme for my site. I always stuck with existing freely-available themes that always left me wanting more in one department or another. Either the typography wasn’t up to par or I didn’t like the comments layout. So, I always had to settle with whatever I could find.

That changed a couple of days ago when I came across a typography post on some blog which inspired me to begin my own wordpress theme. I had one clear goal in mind — improve readability. People come to my site mostly to read the tutorials. It’s not fair if I give text secondary importance focusing on the layout. So, I started customizing the sandbox wordpress theme. It has the cleanest markup and I was able to make all the changes simply through a custom CSS. I went with a fairly large serif font (Georgia) for the content with a sans-serif (Open Sans) font coming from Google Webfonts for the post titles. I also have a slight text shadow effect but it wont’ be visible if you’re using IE. There’s only around 5 images in the whole theme plus two fonts. So, the overall result is a fairly lean page with clear fonts and layout.

As always, all comments and criticism is most welcome.

Varnish Cache for WordPress on cPanel

Varnish is an extremely easy to configure server cache software that can help you counter the ‘slashdot effect’ — high traffic over a small period of time. The way Varnish does this is by sitting between the client and the webserver and providing cached results to the client so that the server doesn’t have to process every page. It’s better than memcache etc because the request never gets to the webserver. You can avoid one of the bottlenecks this way. In this tutorial, we’ll cover how to setup Varnish on a VPS (or dedicated server) where you have root access and are running your site using cPanel/WHM. It also applies to situations where you don’t have cPanel/WHM. You can just skip the cPanel portion if that’s the case. So, let’s get started.

Read more…

Google Has Messed Up Social Once Again

I know I might be in the minority right now but that’s how I feel. It seems Google has learned little from Wave and Buzz. Here’s what I think has gone wrong this time.

First, Google engineers have probably never heard of the phrase, “less is more”. They tried doing everything with Wave — everyone knows how that turned out. They’re doing the same thing with Google+ (or Google Plus). It’s twitter, friendfeed, skype, facebook and slashdot all rolled into one. The problem is, I don’t know which one I’m using when I navigate to the G+ interface.

I know I can divide people into circles and keep them separate but I don’t know if I can keep track of it all. I have separate ‘circles’ for friends and ‘professional connections’. Most often, though, I want to share a thought with both of them so I just post that to the  ’public’ circle. My friends, goofy as they are, start commenting on the post and it quickly turns into a dorm room crap fest. That’s not the ‘professional’ image I want to project — that was the whole point of circles. The solution, post the same thing twice, once to public and again to friends. But then, why don’t I just go over to twitter and post there?

That, I think is the core of the problem. Why would anyone want to use Google+ — after the initial awe of the cool interface for dropping your friends in a circle subsides? For sharing news — I already have a neat little twitter account for that. It’s streamlined and it does what it’s supposed to do. When I’m there, I know what I’m there for. I don’t get distracted by comments from my goofy friends. Well, how about keep tabs on my friends? I don’t use facebook myself but last time I checked a lot of people were already using that social network. Just as people haven’t jumped the Yahoo! mail ship despite the immense impotence of Yahoo!, I don’t see why they’d move everything over from Facebook over to Google+. Not everyone likes to play with new and shiny geek toys.

And that brings me to the second point: Google engineers just can’t shake the geek within them. They think everything will be adopted if it’s similar enough to Gmail. They tried doing this with Wave. They did the same thing with Buzz, integrating it too tightly with Gmail and that was a fiasco. Now, they’re doing this with Google+. It’s all about how cool the technology is. They’re going to release the API soon.  That’s all great but what about the social aspects? I don’t see any incentives for moving away from my existing social networks — except maybe Buzz. So, I don’t think Google+ is a facebook killer or a twitter killer. It might be a  Buzz killer but that too is a maybe.

Arabic/Urdu Typesetting the Easy Way

As the regular readers might know, I’m a big fan and regular user of the LaTeX typsetting system. Keeping to the tradition of occasionally posting a LaTeX tutorial on the blog, here’s one that will be useful to all those that are trying to get up and running with XeTeX — the unicode support adding big brother to the TeX typesetting system. XeTeX basically allows you to use all the Opentype (and other) fonts installed on your system — and it adds unicode support. That means you can use the latest Adobe fonts you’ve got as well as all the Arabic, Urdu, Chinese and Japanese fonts and typeset them as beautifully as you would any Latin script in LaTeX. It’s a hassle to set it up though and you wouldn’t find many tutorials or how-tos on the Internet.

So, here’s one that will allow you to setup and start using XeTeX — especially if you’re trying to find a way to easily typeset Arabic/Urdu. Let’s get started: First, download and install MikTeX 2.7+ (for Windows or any other LaTeX package for your favourite OS). I use 2.8 but I guess you should download the latex (2.9). Everything after 2.6 t has XeTeX built-in. You should download the basic installer from here.

Install also the TeXMaker IDE. We’re going to configure that in a little while to use XeTeX instead of LaTeX. Get the Scheherazade font from the SIL page. You can just extract the zip file anywhere and copy the .ttf file into C:\Windows\Fonts folder. That installs the font required for Arabic typesetting.

Now, fire up TexMaker, go to Options menu and click on ‘Configure Texmaker’. In the Pdflatex section, replace pdflatex with xelatex.

Configure texmaker for XeLaTeX

Create a new file and insert the following code:

\documentclass{article}
\usepackage{arabxetex}
\begin{document}

\begin{arab}[novoc]
mi_tAl: \aemph{45} darajaT
\end{arab}

\end{document}

Before we build, you might need to refresh the font-cache for XeLaTeX (especially if you’ve used it before). For that, go to your MikTeX bin directory and enter the command fc-cache.exe --force. It might take a while but it should run without any errors.

Now, back to Texmaker   and ‘Quick Build’.

Build ArabXeTeX

You should be able to see the output like so:

Arabic Output

During the quick build, you might get a dialog box saying that a certain package is missing and you need to install it. If so, just click ‘Ok’ or ‘Install’ and it will be automatically installed on-the-fly by MikTeX. (You’ll need an internet connection though.) If it simply exits giving some error about a missing .sty file (look at the bottom of the screen for the errors/output messages) — you need to open up the “Package Manager” from the MikTeX folder in the start menu and install that package. For example, if you get the error, “Missing arabxetex.sty”, search for arabxetex package, select it in the list and click on the ‘install button’. Then try to quick build again. Let me know in the comments below if you have any errors and I’ll try to help you out.

Installing MikTeX Packages

Finally, you can look at the original documentation for ArabXeTeX for the details of use. I’ll leave you with another output from the examples in that doc.

Another example output from ArabXeTeX

Readability for the Web

I just came across this great feature in Safari (having downloaded it for Windows 7) that’s called ‘Safari Reader’. It allows you to read articles on any webpage in an extremely readable, uncluttered pane.

Having seen the great utility of this tool, I immediately searched for an equivalent for chrome. Turns out, there’s an extension provided by readability.com that does just that. Reading one of my previous posts using readability, I found it to be a great tool that enhances readability of any page; it’s not only the uncluttered interface but also beautiful typography that will make reading long passages/blogs much much better.

The default pane looks pretty good. The font is modern, yet highly readable on-screen. Click the image below to see full-size snap.

The default, large-font display provided by the Readability.com extension

You can also customize it to give a much more book-like feel. It can turn hyperlinks into footnotes, allows you to re-define the text size and the paragraph width. It also has a couple of pre-defined themes that work really well.

Changing the theme is very easy on Readability extension. Pick a different theme, check the option to convert hyperlinks to footnotes and change the font-size to suit your preferences.

There’s also a wordpress plugin that allows you to let your readers view your posts in readability pane but I didn’t really like that. An alternative to that is to insert custom hyperlinks using the URL shortener rdd.me. You can read this post in readability using this link.

A Basic Naive Bayes classifier in Matlab

This is the second in my series of implementing low-level machine learning algorithms in Matlab. We first did linear regression with gradient descent and now we’re working with the more popular naive bayes classifier. As is evident from the name, NB it is a classifier i.e. it sorts data points into classes based on some features. We’ll be writing code for NB using low-level matlab (meaning we won’t use matlab’s implementation of NB). Here’s the example we’ve taken (with a bit of modification) from here.

Consider the following vector:

(likes shortbread, likes lager, eats porridge, watched England play football, nationality)T

A vector x = (1, 0, 1, 0, 1)^T would describe that a person likes shortbread, does not like lager, eats porridge, has not watched England play football and is a national of Scottland. The final point is the class that we want to predict and takes two values: 1 for Scottish, 0 for English.

Here’s the data we’re given:


X = [ 0 0 1 1 0 ;
1 0 1 0 0 ;
1 1 0 1 0 ;
1 1 0 0 0 ;
0 1 0 1 0 ;
0 0 1 0 0 ;
1 0 1 1 1 ;
1 1 0 1 1 ;
1 1 1 0 1 ;
1 1 1 0 1 ;
1 1 1 1 1 ;
1 0 1 0 1 ;
1 0 0 0 1 ];

Notice that usually when we represent data, we write features in columns, instances in rows. If this is the case, we need to get the data in proper orientation: features in rows, instances in columns. That’s the convention. Also, we need to separate the class from the feature set:

Y = X(:,5);
X = X(:,1:4)'; % X in proper format now. 

Alright. Now, that we have the data, let’s hear some theory. As always, this isn’t a tutorial on statistics. Go read about the theory somewhere else. This is just a refresher:

In order to predict the class from a feature set, we need to find out the probability of Y given X (where

X = ( x_1, x_2, \ldots x_n )

with n being the number of features. We denote the number of instances given to us as m. In our example, n = 4, m = 13. The probability of Y given X is:

P(Y=1|X) = P(X|Y=1) * P(Y=1) / P(X)

Which is called the Bayes rule. Now, we make the NB assumption: All features in the feature set are independant of each other! Strong assumption but usually works. Given this assumption, we need to find P(X|Y=1), P(Y) and P(X).

(The weird braces notation that follows is the indicator notation. 1\{ v \} means use 1 only if condition v holds, 0 otherwise.)

P(X) = P(X|Y=1) + P(X|Y=0) P(X|Y=1) = \prod_j{P(x_i|Y=1)}

To find P(X|Y=1), you just have to find P(x_i|Y=1) for all features and multiply them together. This is where the assumption comes in. You need the assumption of independence here for this.

P(x_i|Y=1) = \sum_j{1\{x_i^j = 1, y^j = 1\}} / \sum_j{1\{y^j = 1\}}

This equation basically means count the number of instances for which both x_i and Y are 1 and divide by the count of Y being 1. That’s the probability of x_i appearing with Y. Fairly straight forward if you think about it.

P(Y=1) = \sum_j{1\{y^j = 1 \}} / \sum_j{1\{y^j = 1, y^j = 0 \}}

Same as above. Count the ratio of Y=1 with the total number of Ys. Notice that we need to calculate all these for both Y=0 and Y=1 because we need both in the first equation. Let’s begin from the bottom up. For all of below, consider E as 0 and S as 1 since we consider being Scottish as being in class 1 (positive example).

P(Y):

pS = sum (Y)/size(Y,1);     % all rows with Y = 1 
pE = sum(1 - Y)/size(Y,1);  % all rows with Y = 0

P(x_i|Y):

phiS = X * Y / sum(Y);  % all instances for which attrib phi(i) and Y are both 1
              % meaning all Scotts with attribute phi(i)  = 1 
phiE = X * (1-Y) / sum(1-Y) ;  % all instances for which attrib phi(i) = 1 and Y =0
              % meaning all English with attribute phi(i) = 1 

PhiS and PhiE are vectors that store the probabilities for all attributes. Now that we have the probabilities, we’re ready to make a prediction. Let’s get a test datapoint:

x=[1 0 1 0]';  % test point 

And calculate the probabilities P(X|Y=1) and P(X|Y=0)

pxS = prod(phiS.^x.*(1-phiS).^(1-x));
pxE = prod(phiE.^x.*(1-phiE).^(1-x));

And finally, the probabilities of P(Y=1|X) and P(Y=0|X)

pxSF = (pxS * pS ) / (pxS + pxE)
pxEF = (pxE * pS ) / (pxS + pxE)

They should add upto 1 since there are only two classes. Now you can define a threshold for deciding whether the class should be considered 1 or 0 based on these probabilities. In this case, we can consider this test point to belong to class 1 since the probability pxSF > 0.5.

And there you have it!

ProFTPD with SFTP, MySQL and Monthly Usage Reports

To install ProFTPD with MySQL-based authentication and SFTP support, you need to download the latest version of the source code and build it with customized options. But first, some prerequisites. (These instructions are for ubuntu but can easily be modified for CentOS).
Read more…

Enabling Voicemail for A2billing

So you’ve setup a2billing and have everything working out, all the DIDs are forwarded to a2billing and all the call plans are set. All you’re missing is a voicemail system. Wait no more. Here’s how to enable voicemail for a2billing users.

First, you need to change the A2billing class to enable it to forward unavailable or unattended calls to the voicemail. For that, edit the [a2billing-home]/common/lib/Class.A2billing.php file file. The two functions that need changing are the call_did and call_sip_buddies. In version 1.8.5, the changes are around line 1160 and 1330. You can just search for the following code to reach there. (Remember, there are two instances that need changing.)

//# Ooh, something actually happend!
if ($dialstatus == "BUSY") {

Now, it needs to be changed so that the caller is redirected to the voicemail if the dialstatus is CHANUNAVAIL, CONGESTION or NOANSWER. Note that we need to get rid of the existing checks for these statuses and introduce our own code. The complete code would be as follows:

$answeredtime."-DIALSTATUS=".$dialstatus."]"); 
$lang = "en";
if (($dialstatus =="CHANUNAVAIL") || ($dialstatus == "CONGESTION") ||($dialstatus == "NOANSWER") || ($dialstatus =="BUSY") )
{
	// The following section will send the caller to VoiceMail with the unavailable priority.
	$did_ext_number = substr($this->destination, 4);
	// get the actual did 
	// BAD but a hack for now 
	$db_host = $this->config['database']['hostname']; 
	$db_user = $this->config['database']['user']; 
	$db_pass = $this->config['database']['password'];
	$db_sele = $this->config['database']['dbname'];
	
	mysql_connect($db_host, $db_user, $db_pass);
	@mysql_select_db($db_sele) or die( "Unable to access database");				
	$query = "select S.name, D.did, C.language from cc_sip_buddies S, cc_card C , cc_did_destination E, cc_did D 
				where S.id_cc_card =C.id and E.id_cc_card = C.id and E.id_cc_did = D.id and 
				S.name =  '$did_ext_number';";
				
	$result = mysql_query($query);
	$did_number = $did_ext_number; 
	if ($row = mysql_fetch_array($result)){		  
		if($dialstatus =="BUSY")
			$did_number = "b".$row["did"]; 
		else
			$did_number = "u".$row["did"]; 
		
		// set the language 
		$lang = $row["language"]; 
	}
	
	$this -> write_log("[STATUS] CHANNEL UNAVAILABLE - DIVERT TO VOICEMAIL ($did_number)");
	$lang_str = "LANGUAGE()=$lang";
	$agi->exec (Set, $lang_str); 
	$agi-> exec(VoiceMail,$did_number);
} 


// the old code for BUSY is commented out 
//# Ooh, something actually happend!
/* if ($dialstatus  == "BUSY") {
	$answeredtime = 0;
	if ($this->agiconfig['busy_timeout'] > 0)
		$res_busy = $agi->exec("Busy ".$this->agiconfig['busy_timeout']);
	$agi-> stream_file('prepaid-isbusy', '#');
} elseif ($this->dialstatus == "NOANSWER") {
	$answeredtime = 0;
	$agi-> stream_file('prepaid-noanswer', '#');
} else
*/ 

if ($dialstatus == "CANCEL") {
	$answeredtime = 0;
} elseif ($dialstatus == "ANSWER") {
	$this -> debug( DEBUG, $agi, __FILE__, __LINE__, "-> dialstatus : $dialstatus, answered time is ".$answeredtime." \n");
} elseif ($k+1 == $sip_buddies+$iax_buddies) {
	$prompt="prepaid-dest-unreachable";
	$agi-> stream_file($prompt, '#');
}

/* 
// AGAIN, the old code for the statuses is commented out 
if (($dialstatus  == "CHANUNAVAIL") || ($dialstatus  == "CONGESTION"))
	continue;
*/ 

Find the similar code in the other function and change that too.

Now, we need to change the asterisk voicemail configuration to recognize the DID as a mailbox. For that, first enable asterisk realtime. (It’s a simple matter of compiling asterisk and then compiling asterisk-addons.) After that is done, change the /etc/asterisk/res_mysql.conf file as follows:

[general]
dbhost = 127.0.0.1
dbname = a2billingdb
dbuser = [asteriskuser]
dbpass = [yourpassword]
dbport = 3306
dbsock = /tmp/mysql.sock

Also, in the /etc/asterisk/extconfig.conf add the following line:

voicemail =>mysql,a2billingdb,voicemail_users

This will make sure that the voicemail application tries to find the list of mailboxes in the voicemail_users table in the a2billingdb. Let’s create that table:

CREATE TABLE `voicemail_users` (
`uniqueid` int(11) NOT NULL auto_increment,
`customer_id` int(20) NOT NULL default '0',
`context` varchar(50) NOT NULL default '',
`mailbox` varchar(20) NOT NULL default '0',
`password` varchar(20) NOT NULL default '8888',
`fullname` varchar(50) NOT NULL default '',
`email` varchar(50) NOT NULL default '',
`pager` varchar(50) NOT NULL default '',
`stamp` timestamp(14) NOT NULL,
PRIMARY KEY (`uniqueid`),
KEY `mailbox_context` (`mailbox`,`context`)
) TYPE=MyISAM;

… and synchronize it with the rest of the a2billing database so that the users can use their DID and SIP secret for their mailboxes.

insert into voicemail_users(customer_id,context,mailbox, password, fullname, email)
select S.id_cc_card, 'default', D.did, S.secret, concat(C.lastname,' ',C.firstname) fullname, C.email
from cc_sip_buddies S, cc_card C , cc_did_destination E, cc_did D
where S.id_cc_card =C.id 
and E.id_cc_card = C.id 
and E.id_cc_did = D.id;

You might want to add the following cronjob to perform this synchronization automatically.

0 * * * * mysql -ua2billinguser -p[yourpassword] -D a2billingdb -e "truncate table voicemail_users; insert into voicemail_users(customer_id,context,mailbox, password, fullname, email) select S.id_cc_card, 'default', D.did, S.secret, concat(C.lastname,' ',C.firstname) fullname, C.email from cc_sip_buddies S, cc_card C , cc_did_destination E, cc_did D where S.id_cc_card =C.id  and E.id_cc_card = C.id  and E.id_cc_did = D.id ;"

That should fix the whole voicemail recording business. However, we still need to enable the voicemail interface provided by FreePBX. Since the ARI does not support asterisk realtime, we need to change the login.php file in /var/www/html/admin/includes folder. Search out the code where the authentication is being done. The code first tries to autenticate using the configuration file. It then tries SIP authentication (whatever that is). After that, we insert our own DB authentication like so:

// check database login:  recly 
if(!$auth){
	// BAD but a hack for now 
	$db_host = "localhost"; 
	$db_user = "a2billinguser"; 
	$db_pass = "[yourpassword]";
	$db_sele = "a2billingdb";

	mysql_connect($db_host, $db_user, $db_pass);
	@mysql_select_db($db_sele) or die( "Unable to access database");

	$query = "select * from `voicemail_users` where mailbox = '$username' and password= '$password'"; 
	$result = mysql_query($query);

	if ($row = mysql_fetch_array($result)){		  
		$auth = true; 
		$extension = $row["mailbox"]; 
		$outboundCID = "";
		$displayname = $row["fullname"];
		$vm_password = $row["password"];
		$category = "";
		$context = $row["context"]; 
		$voicemail_enabled = "1";
		$voicemail_email_address = $row["email"];
		$voicemail_pager_address = $row["pager"];
		$voicemail_email_enable = "yes";
		$voicemail_email = array('');
		$default_page = $ARI_DEFAULT_USER_PAGE;
	}
}
// the original login failure lines follow 

// let user know bad login
if (!$auth) {
	$_SESSION['ari_error'] = _("Incorrect Username or Password");
}

And that should enable you to login using your ARI. One final thing: You might also want to enable your users to check their mailbox. For that, change the dialplan in extensions_a2billing.conf:

[a2billing]
exten => 9999,1,VoicemailMain()
exten => _X.,1,Answer
exten => _X.,n,Wait(1)
exten => _X.,n,DeadAGI(a2billing.php|1)
exten => _X.,n,Hangup

Now your users can dial 9999 and check up on their voicemail.

Helpful Tips for Newbie System Admins

Before you start reading this tutorial, let me remind you once again who the intended audience of this post is — newbie system administrators. If you’re an experienced admin and are going to laugh at my naivete for writing such basic stuff, please go away — or provide some more ‘advanced’ tips in the comments below so that I know better for the future.

Anyway, let’s begin. If you’re like most newbie system admins, you run a windows system on your home PC / work laptop and connect to the servers you’re managing using putty. Everyone uses putty, you say. Well, yes but there are alternatives. One of the things I hate is to have to copy/paste all the usernames and passwords into that putty session before I can login. So, let’s get rid of that first.

Read more…