Lies and Statistics

Having operated a service business for almost three years, one of the most important lessons I’ve learned is that you can’t control your business results if you don’t know your business results.  I didn’t start with standard business software, preferring to create my own custom solutions as each need arose.  I built spreadsheets as I went along in order to track such things as receipts, employee costs, employee productivity, fixed costs, advertising costs, time spent on projects, and so on, and to analyze these results for trends.  Every time I wrote a new spreadsheet, my eyes would pop open with a new revelation or epiphany which usually translated into a concrete business decision, one that was often a radical departure from previous practices.  Through many incremental steps, and after many spreadsheets, the business which was at one point losing $2000 per month, is now profitable enough to sustain my family.  That is a tremendous accomplishment which would have been impossible without access to good information and analysis.  Data and statistics are absolutely critical to great success in any long-term venture (or adventure).  Wrong information invariable produces wrong decisions.

Ever since I started maintaining this website again, I have been interested in knowing whether anyone was reading it or not.  There are a variety of statistics programs out there which all use slightly different algorithms and reveal slightly different results.  The Yahoo webhosting account page has a rudimentary bar-chart which shows number of hits.  The Wordpress blog account page had a similar chart, but I downloaded a plugin called Statpress which gives more detailed results.  I have had two problems with using Statpress.  First, it only shows blog article hits; it doesn’t account for hits to any of the html pages which make up a large part of the website.  Second, for some inexplicable reason (which I can find no support for), it usually doesn’t load in my browser.  Instead, it causes my browser (Firefox) to download an empty file called index.php, and then open an endless recursive stream of empty, untitled pages until I finally kill it.  That is what it usually does.  Occasionally, it actually opens like it is supposed to.

I started using my Yahoo webhosting account to create access log files.  These are text files where every line shows all the available information about a single hit from a single visitor.  50,000 hits– 50,000 lines of text.  Yahoo has a link to a free weblog analyzer called “Weblog Expert Lite.”  It is a slimmed down version of a presumably more expansive program which you have to buy.  I started using the Lite version to track hits via my access log file.  It gave me more optimistic numbers than the other charts I had.  But it didn’t filter out my own hits, or spider or robots hits (the creepy-crawlers on the web that index everything in the background.)

When I put Google ads on my site, I signed up for a Google Analytics account.  To use it, I insert java code into every page that I want to track and when you load a page, the script executes a command that sends information to my Google account, which then creates a statistical analysis of the traffic.  The Google Analytics software gave very pessimistic results.   The Weblog Expert Lite showed 1,000-3,000 hits per day, and the Google software showed about 75 page views per day.  I concluded that either my website was still more niche than the Weblog Expert indicated, or else I was getting a lot of traffic that Google wasn’t giving me credit for.  I just had to know.

I was on the verge of writing my own spreadsheet to analyze the access logs, when about a month ago I found an access log analysis program called AWSTATS.  It is open-source, with a primary version for Linux.  I tried installing it then, but success would require more time than I had available at the time.  Yesterday I did a full successful install.  It turned out I had to first install the latest version of Perl.  I know about as much about Perl as is necessary to install it.  It appears to be a programming or scripting environment.  I finally got my AWSTATS program to work.  It creates a static webpage full of charts based on the access log file and a configuration file.  It is the most powerful statistics program I have seen so far.  That power comes from the configuration file, which is 1500 lines long (relatively long explanatory paragraphs, each followed by a single-line configuration setting that is user-definable within the given parameters).  There are so many settings you can change that the abilities of the program to show you what you want to know seem limitless.  Writing my own spreadsheet would take a very long time and probably not have as many functions as this program does, and would probably require more computing resources than AWSTATS does.

Here is what AWSTATS told me about my website.  The rate of growth that the other programs showed me is relatively accurate, at least in terms of page views and hits.  My hits and views have doubled since I started maintaining the site again.  Cool.  Actual visitor growth is slower than the other programs report.  Visits have grown by about 50%, while unique visitors have not grown much at all.  That means perhaps that the same people read my site as always have, but they are finding more to read.

What was most revealing is that according to AWSTATS, 70% of my site traffic can be accounted for by my own visits and spider visits.  That means that this website is only 30% as popular as the free Weblog Expert Lite program suggested.  That is a bit humbling, but it puts traffic in perspective.  I am keeping the spiders happy, and creating value for myself, definitely.  Beyond that, the website is still very niche.  That means that I am still writing for a fairly intimate audience, along with surfers who are possibly looking for something entirely different.  Having finally gained access to better data, I can start to think intelligently about what audience this site might have and how to serve that audience with relevant content.  Knowledge is power, and good information is key.