Web Analytics and Business Intelligence — Marriage Made in Heaven, Why Still Separate?

Not long before I was given an opportunity to present at the 2006 eMetrics conference in DC, I had been promoted to manage the combined web and business intelligence teams at USAC. The reason was simple — management by metrics was the only way to go if this non-for-profit organization responsible for the $8 billion Universal Service Fund was to be efficient. My thinking then was that web analytics and business intelligence were complementary. Today, my thinking is that they should not be viewed as separate disciplines anymore.

Web Analytics and Business Intelligence — Marriage Made in Heaven

The days of the separate tools are numbered. Big data vendors left and right are offering ways to integrate data from disparate sources. The skill sets should not be unique anymore as we are all to analyze the ways for optimizing an organization’s engagement with customers and constituents as well as its operational efficiency.

Web Analytics and Business Intelligence -- marriage made in heaven
Web Analytics and Business Intelligence — marriage made in heaven. Image courtesy of IQWorkForce.

If Web Analytics is Business Intelligence, as the DA blog claims, why do we still view them as separate disciplines? If eConsultancy is inviting us to break down the silos between Web Analytics and Business Intelligence, why do we still cling to these obsolete boundaries?

True, there is data that originates on websites and data that originates in transaction systems but data is data. Do we need to split mobile analytics from web analytics? What about piping data from the internet of things? It is time this false dichotomy be put to rest.

What we need are skilled data analysts and flexible data analysis tools!



Digital Analytics Helps You Measure Your Website Objectives

Anybody who has a small business website should sign up to this fabulous and free online course on digital analytics which will help us all take advantage of the enormous power of Google Analytics.

Digital Analytics tools like Google Analytics Help You Measure Your Website Objectives
Digital Analytics tools like Google Analytics Help You Measure Your Website Objectives

What is Digital Analytics, a.k.a the evolution of Web Analytics

In the definition of the best story teller among web analytics evangelists, Avinash KaushikWeb Analytics 2.0 is:

(1) the analysis of qualitative and quantitative data from your website and the competition,
(2) to drive a continual improvement of the online experience that your customers, and potential customers have,
(3) which translates into your desired outcomes (online and offline).

Addressing the importance of digital analytics, this informative video guides us to address the type of customers we serve and the business goals we can address through the power of digital analytics.

Know Thy Site: What are your website objectives and how can you measure them through digital analytics?

“If one does not know to which port one is sailing, no wind is favorable.”
~ Lucius Annaeus Seneca

According to Google’s Digital Analytics Academy, the five common business objectives for websites are:

  1. First, for ecommerce sites, an obvious objective is selling products or services, for ecommerce websites.
  2. Second, for lead generation sites, the goal is to collect user information for sales teams to connect with potential leads.
  3. Third, for content publishers, the goal is to encourage engagement and frequent visitation.
  4. Fourth, for online informational or support siteshelping users find the information they need at the right time is of primary importance.
  5. And finally for branding, the main objective is to drive awareness, engagement and loyalty.

I decided to use the list above and determine the type of business objective for some of the websites I manage either personally as a blogger, collectively  with my colleagues on behalf of my employer, or as a website optimization consultant on behalf of my clients. As it turns out, I have quite a variety of websites with diverse business objectives:

Taking the time to reflect on the type of objective applicable to a website is deeply satisfying — a delight for geeks, I would say — because it sets one on the course of actually managing, rather than just maintaining a website. And there is no better tool for managing through measurement than digital analytics. Peter Drucker would be delighted with the power of the tools we have easily accessible today!

New website for Slow Food DC

Today, after a very fruitful collaboration with a talented team of volunteers, I launched the newly redesigned website for Slow Food DC, which is the Washington, DC metropolitan area chapter of Slow Food. It’s mission is very worthwhile, “Supporting Good, Clean, and Fair Food”, and I was delighted to assist them.

Slow Food DC

I got involved with this project after an old friend of mine, Alexandra Greeley, suggested my services to the chair woman of Slow Food DC, Kati Gimes. Before you know it, I was consulting with a group of young dedicated volunteers, many of whom have done wonderful work in website redesign or social media management for other DC-based non-profits, themselves.

During our meetings, we discussed the content strategy, the target audiences, the information architecture, and the technological platform. A young Photoshop aficionado took upon herself to design the banner incorporating the new Slow Food DC logo.

The site uses the new WordPress theme twentyone which is extremely powerful. It utilizes plug-ins for Google Analytics, Twitter and RSS feeds. It truly is a marvelous content management system which will burden of needing a webmaster any time a new content update is needed.

Congratulations, Slow Food DC! Cheers!

Opting out of behavioral advertising

Today I was reading an article on NYTimes.com and noticed on the side a list of additional suggested reading for those interested in non-profit management, courtesy of LinkedIn. Of course this can be valuable but the fact that LinkedIn shares information about my employment with NYTimes was nothing I had considered when signing up with LinkedIn years ago.

If you find this, or other similar scenarios annoying, here is a valuable tool to opt out of behavioral advertising networks, link courtesy of Jim Stearne!

Demographic predictions on the web and the Baha’i Faith

Microsoft recently released a series of web analytics tools that dip into the database of MSN users and mashes demographics with predictive analysis. For example the following tool predicts the demographic distribution of U.S. users of the MSN Search engine.

I am a Baha’i and I was curious to see what this new tool has to say about Baha’i-related searches. I tried several different queries – you might find these results interesting (keep in mind the data comes from the U.S. only):

Among the teachings of the Baha’i Faith are the equality of women and men. Yet the searches for “Baha’i Faith” (note the correct use of apostrophe) on MSN got predominantly female audience:

Query: baha’i faith

Gender: Female-oriented, with the following confidence:
Male :0.27
Female :0.73

Searches for “Bahai Faith” (without the apostrophe) got a more balanced audience (probably, a proof that women are better spellers 🙂 :

Query: bahai faith

Gender: Female-oriented, with the following confidence:
Male :0.46
Female :0.54

Continuing with the principle of three onenesses: the oneness of God, oneness of religion and oneness of humanity, yield intriguing results: searches for one humanity, one world, one religion, one faith are all predominantly female but apparently the searches for one god are heavily male. You gotta love web analytics 🙂

Query: one humanity

Gender: Female-oriented, with the following confidence:

Query: one world

Gender: Female-oriented, with the following confidence:

Male :0.41
Female :0.59

Query: one religion

Gender: Female-oriented, with the following confidence:
Male :0.31
Female :0.69

Query: one faith

Gender: Female-oriented, with the following confidence:
Male :0.31
Female :0.69


Query: one god

Gender: Male-oriented, with the following confidence:
Male :0.73
Female :0.27

Here is for those curious to learn more about the Baha’i Faith

The Executive’s Guide to Web Site Measurement and Testing

I first saw Phil Kemelor talk at the Web Managers Roundtable in December 2006 at the National Building Museum. After that he contacted me, referred by Julie Permutter, to participate in a survey on the challenges of web analytics implementation. Following the survey, we decided to meet for lunch which turned into a very pleasant discussion on a range of web management and analytics related issues. Phil is a very friendly, pleasant and knowledgeable fellow who is deeply passionate about making organizations more effective through maximizing the return on their web investments.

Phil has published recently his own “Executive’s Guide to Web Site Measurement and Testing”. Phil’s contributions to the web measurement field are based on his work in web analytics implementations since the late 1990s, so he does bring experience and not just theory while sharing his findings succinctly in a book that targets not the techies but executives who are interested in tapping the marketing potential of their web channel. Here are the main themes of his book:

The first part of the book explains why you need web site measurement to boost Web site profitability, defines site measurement objectives and show you how to plan a successful program.

The second part of the book focuses on conducting site analysis, developing metrics, and analyzing data so that you can make the site design, marketing, and business decisions that will increase Web site profitability and effectiveness.

The final, third, part of the book focuses on what it takes to implement web site measurement and testing for an organization’s website.

I will be writing a proper review of Phil’s book later on. In the mean time, if you want to catch Phil talk on web analytics, he is scheduled to lead a series of seminars on eCommerce and Measurements in the coming months, as part of the next stage evolution of the Web Managers Roundtable.

WebTrends Training for Technical Professionals – Day 4: Advanced Reporting – Drilldown Reports

Last day of my WebTrends training. We continue with Custom Reports and will focus today on drill down reports.

With parameter analysis, we are limited to two parameters; with custom reports, also limited to two parameters. Drilldown dimensions shows you all the dimensions you would want to track down and can select for your custom report.

Another use for drilldown reports is when you have a lot of external information and you want to put it together. There is a pre-defined hierarchy for campaign attributes in WebTrends.

Using SmartView to track customer behavior

SmartView integrates actual view of the website in a browser with the WebTrends data. It is a great tool for novices to web analytics to give them a sense of what information is available.

For some of the functionality of SmartView you need to use session or persistent cookies or user ID and password — IP / User Agent sessionalization is not sufficient. Also, visitor history needs to be enabled.

SmartView compares the URI on the link to data. SmartView does not work with Javascript / pull-down menu links.

If you have two links on a page going to the same destination, i.e. top link and bottom link, you need to add the following parameter WT.svl to each link, i.e.


This would split the reported traffic between the unique links.

Install SmartView on each user’s machine and give them view access to SmartView. Access is based on profile basis.

Advanced SmartView reports are based on “page transitions”. These are combination of the page the visitor was on and the next one the customer visited.

WebTrends Training for Technical Professionals – Day 3: Advanced Reporting

Today is the third day of my WebTrends training and the first day of the advanced reporting training.

We will focus on :

– visitor history table usage in WebTrends
– configuring campaign tracking
– defining new dimensions, measures, filters to create custom reports, etc.

Clearing the confusion of hit metrics and visit metrics:
– Log file with 1 visit consisting of 3 page views, where each page view was to a different content group, nevertheless there is only one visit (although it was a visit to three different content groups)

Visitor History

To use visitor history you need to use persistent cookie, session cookie or persistent authentication (user login). By default, visitor history is not enabled and can be enabled by clickin on the Visitor History tab of the profile overview. If configurning the profile to export the visitor history, it would get stored in a specified directory on a daily, weekly or monthly basis.

Visitor history can include the following behavior categories:

– Campaign History
– Search Engine History
– Most Recent Search Engine Duration
– Visit History
– Purchase History
– Custom Visitor Segmentation
– Content Group Unique Visitor Tracking – new to 7.5
– Page of Interest Unique Visitor Tracking – new to 7.5

You can invent your own segmentation through tagging. Unique visitor would translate visits to visitors. The precision of unique visitor tracking is based on the cookie. The same visitor using different browsers from the same PC would be counted as two different unique visitors.

Since visitor history might grow, trimming can be handy. Visitor history trimming is based on several criteria — number of visits, lifetime value, most recent visit, most recent purchase, etc.

Visitor history gets updated based on the profile. If you want retroactive data you need to reanalyze.

Visitor history is important as a tool for visitor retention. It is cheaper to keep a customer happy than to get a new customer.

In version 8, Marketing Lab would enable the mashing of web log data and additional customer relationship information.

Visitor history can be used to understand visitor behavior:
– on which visitors should you spend marketing dollars?
– should some visitors be contacted more often than others?
– which campaigns generate the best visitors, etc?

Visitor History Table

– Visitor cookie ID (based on cookie sent by SmartSource or your own cookie parameter)
– Contains attributes on a per visitor basis:
– number of hits
– number of visits
– time of first visits
– time of last visits, etc.

Visitors tracked by IP address/User Agent will not be stored in the Visitor History Table.

Consider if you need to use visitor history because it slows down reporting and takes hard drive space. Try it by turning it on for a couple of profiles and see if you would get value from it.

Measuring Campaign Performance

To track paid (vs. organic search engine keywords), use the following tags:
WT.mc_ID=”marketing campaign code”&WT.srch=1

WT.srch=1 would automatically put the data into the paid-search category reports.

When tracking campaigns from multiple channels, the easiest is to have unique campaign ID. Wrap the campaign ID translation table which is limited to 26 columns (in custom reports). Why 26, because each column in Excel is associated with a letter. Possible values are:
– campaignID
– description
– offer
– creative type
– channel

The canned WebTrends campaign report uses 11 columns in the translation table. For the columns that are not used, you can put commas in the columns with no data.

The prefered method for tracking each campaign in WebTrends is to use the WT.mc_id parameter.

META tags:
1. Unique landing page (tagged page)
2. Pass WT.mc_id as part of the page URL

META tag syntax:

The META tags have to be above the SmartSource javascript code.
Campaign ID is a visit parameter.

Custom Reports

Custom Reports consist of one table and its associated graphs. Once created, reports are automatically added to:
– custom reports main screen
– custom report tab on profile
– reports pick list in Dashboards
– report pick list in Templates

Custom Reports can have a primary and secondary dimension, i.e. domains and URL with Query String. You can have up to 20 measures.

To create a Custom Report:

  • Define the components:
    1. Data sources first
    2. Dimensions
    3. Measures
    4. Filters
  • Custom Report
  • Add the Custom Repor to profile

Translation files are easy but the downside is, they get stale. It is OK to sue translation table for things that would not change, for product categories for example, but for more dynamic information, like pricing, not.

Data Sources are associated with Dimensions and Measures.

When creating a Custom Report, make sure to check the box for “Exclude traffic where this dimension was unspecified”, otherwise the report would include those that did not mean the criteria for the dimension.

In a basic report you cannot turn on the Custom Report. Make sure to select advanced profile for custom reports.

Important Tip: To make sure that the Custom Reports show in every report, go to Report Designer > Templates and select to edit the Enterprise Complete View report. Click the Content tab and then click on the Enterprise Complete View V7.5 item on the left navigation. Add a new report and select from the Custom Reports list the following one: “Add a folder to my template that will include all current and future Custom Reports.” When you click on it, you will see “Auto-Populated at Report Time”.

WebTrends Training for Technical Professionals – Day 2

Visitor History is a nice new feature of version 7.5 and up of WebTrends. It requires strong authentication, i.e. session cookies, persistent cookies but not IP address & user agent. Don’t turn it on if using IP address for authentication because it does make the profile larger and slows them down.

Running three segments of large amounts of data of 10 days each would run generally faster than 10 segments of 3 days each. But make sure you have the RAM and the hard drive in place before running large amounts of traffic.

Page files and download files can be defined on per profile basis. For example Flash files can be both downloads and page views. By default they would be under page views but it makes sense to configure the profile and assign swf (Flash files) as download files.

In Reports > Reports Headers you can customize the logos of the banner and replace the WebTrends Report Banner with custom images. The middle image is a spacer and it tiles it, so if the logo is large, do not use the middle image.

In Reports / Reports Types, turning all five reports types make sense — Yearly, Quarterly, Monthly, Weekly, Daily. If the reports start on June 1st, clicking on Quarterly, does not mean it will generate Quarterly reports for the first two quarters.

Parent-child profiles split your web traffic based on configurable rules, and create multiple report sets. This is useful for service providers and large enterprises. For example, a split based on domain can create a separate report for each domain.

When WebTrends analysis a log file it looks at the first hit, last hit and file size to compare and check whether it had run a report already.

Have a log file directory which keeps the recent log files.
Current directory for storing 1 – 30 days.
Another directory for storing 1 month – 6 months.
Another directory for storing older than 6 months.

Another approach is to use date macros in the log file paths:
ex%date-1%%yy%%mm%%dd%.log – for yesterday’s files
ex%date-2%%yy%%mm%%dd%.log – for 2 days ago
ex%date-3%%yy%%mm%%dd%.log – for 3 days ago

This always checks the last three days.

For special request profiles, the recommendation is to use a separate data source.

New to WebTrends 7.5 is the ability to schedule the download of reports. The smart thing to do is to design the template for the reports they need to see on a regular basis. In Scheduler > Scheduled Events > Schedule Report > Select profile, etc. Select a custom data range for the reports to be emailed.

Clear Analysis Data is different from Reanalyze. Make sure to have the new data source or editing the data source so reanalyzing would be meaningful.

To track the last three days of the SDC log files, use the following macro where sdc is the unique sdc identifier (usually 20 characters) if SDC stores logs on a daily basis:
sdc_%date-1%%mmm%_%dd%_%yyyy%.log – for yesterday’s log file
sdc_%date-2%%mmm%_%dd%_%yyyy%.log – for the day before yesterday
sdc_%date-3%%mmm%_%dd%_%yyyy%.log – for two days before yesterday

Session tracking: visits vs. visitors

Visit — all hits for any one visit
Visitor – anyone entering the web site

Sessionalizing by persistent cookie is considered more reliable than IP address & user agent. However, the same person visiting from work and home would be counted as two separate visitors.

To setup session analysis go to Web Analysis > Session Tracking. For example to track by parameter, we would use Parameter Match. WT scrolls you through grayed-out methods that are not used. Then in track sessions by parameter you would specify the parameter name.

If switching from 3rd party to 1st party cookie, you must redeploy the new tag to all your pages. Identify fallback method, most likely IP address and user agent.

Filters: All profiles by default come with all include filters. There are include and exclude filters. If there is not filter, all activity is included. There is no limit to the number of filters per profile.

To exclude spiders, create a new hit filter. From the browser drop down menu, select All Spiders and Robots [allspiders] which would include the browser.ini file which includes over 500 spiders.

Filters work from now on.

To analyze all campaigns, listing the Campaign ID in the profile would give you only number of visits but not a drill-down report. For “butterfly”-type drill-down report, create a separate profile filtering all entries from ?WT.cm_ID=*

Creating an include filter for the custom 404 files only would give a list of most failing pages which would be very valuable for web developers.

Multi-homed domain log file includes multiple data sources.

Remember, OR is more, AND is less.

If you are to email business users the reports they want to see, find out what they want and put it in a template.

Notes from the WebTrends Training for Technical Professionals

I am attending WebTrends classes for technical professionals and will share some of my notes taken durnig the classes. The notes will not be complete nor very eloquent but will capture the most important information during the training.

Notes from day 1: WebTrends Essentials for Technical Professionals

Content groups can be defined two ways: through content words contained in URLs or through page tagging.
The page tag for content groups is WT.cg_n:
Set meta “WT.cg_n”=content “keyword” – for content groups (URL+WT.cg_n=”keyword”)
Set meta “WT.cg_s”=content “subkeyword” – for sub content groups

WebTrends by default truncates query parameters.

The report graphs and dashboards require the installation of Java which would require one to be administrator on their PC.

Daily reports saves hourly traffic. It makes sense to keep the daily reports for not more than 3 – 6 months.

Custom calendar range used daily reports. Comparative calendar range uses weekly reports.


WebTrends would look at it as three different page views although it is the same page. You would need to build either custom report or parameter analysis to avoid the URL truncation.

You can use translation table mapping query IDs to titles of the articles published.

Path analysis can give you view into 20 steps before and 20 steps after a defined URL. Scenario analysis would have the actual URLs and query parameters for each step and view the path analysis with query parameters.

In a form with multiple fields, you can tag each field to create a scenario analysis with one page only but multiple fields using the on-click event.

In the WebTrends Resource Center, very valuable information is in “Understanding Data Collection” under Implementation > “WebTrends Query Parameters” and “Tracking Downloads and Form Completions”.

With SmartSource Data Collection information about PDF downloads is not logged. You need to do on-click event to track PDF download.

To track Google paid search results, use the following tags:

WT.mc_ID=10001&WT.srch=1 where 10001 is the marketing campaign ID and WT.srch=1 tells WebTrends this is paid search and automatically updates paid search terms and phrases.

WebTrends cookies by default are set as persistent cookies with 10 year expiration. Cookies are the primary identifier for visitor history.

WebTrends_ID cookie contains customer IP address, time in milliseconds since 1974 and duration.

Rejection for 1st party cookies are around 12%, 3rd party cookies around 25%. Home users of AOL readily accept cookies in IE but Firefox does not accept 3rd party cookies by default.

Thirty minutes of inactivity ends a user session. Multiple users through a corporate firewall would generate a single session as long as there is less than 30 min between visitors. AOL can overstate page views by assigning randomly a shared IP address.

Usually traffic measurements based on cookies is better by 20 – 30%.

1st party cookies means that my site is issuing the cookie.

Set in the profile to save 180 daily reports and delete the older reports — rolling 180 daily reports. The feature to remove old files is disable by default.

Deleted Data Cleanup is under Scheduler > Scheduled Events. To enable the feature, unclick the check box for disabling the feature.

Keep quarterly and yearly reports for ever. Keep daily reports only for the last 6 months.

Standard size for a profile is about 1 MB per day. Recommendation no more than 30 custom reports per profile.

Version 8 will allow you to define what is an event. You can define what events you want to scoop out of the log file and automatically export into Oracle or SQL server and run queries there.

The proprietary database is non-querable; in version 8 you can export it for queries.

Comparative calendar view would compare week vs week or month vs month — likewise ranges.

Important: For custom calendar range the dates need to be contiguous. Visitors summary would show incorrect visitors summary in a custom calendar range because it sums up the daily visitors for each day into a total numver for the custom calendar range. You would need to reannalyze the data for the custom period if you want the total number of visitors.

Webtrends tracks annual visitors based on calendar year. So a visitor in December 2005 and a visitor in January 2006 would be consider as different even if they use the same IP address, same browser, same cookie.

Custom comparative view would allow you to compare two custom periods.

All custom reports add only dailies.

A template is a collection of reports — i.e. executive, marketer’s and webmaster’s reports. The dashboard always starts with the default template but I can choose a different template if given access to more than one templates.

Dashboards are not exportable. Reports are exportable into Word, Excel, PDF, or CSV format. You need to install the WebTrends Report Exporter.

Recommendation: Allow sharing bookmarks only among people who are knowledgable about WebTrends to avoid having the bookmarks list be longer than the content list.

You can bookmark queries.

By default of SDC rotates log files hourly, meaning it creates 24 files per day; the recommendation is to change to daily.

Create unique data source per server but use more than once data source per profile. It is better to read the SDC log files from the WebTrends server rather than from the SDC server.

WT.sdl is used exclusively to track multiple links from the same page, i.e. WT.sdl=top would be added to the top link and WT.sdl=bottom would be added to the bottom link.