LongTailMiner v0.1 alpha – find long tail keywords nobody thought about

I’m really enjoying this blogging thing! Every comment I am getting from my readers is a new idea that I feel rushed to put into practice.

My reader, Andrea, mentioned she parses log files to mine for keywords as well. That is an excellent idea.

I decided to put that idea into code and here is a new tool to mine for long tail keywords.

To make really good use of it, I would setup a PPC campaign in Google with a “head keyword” in broad match, bidding at the minimum possible. Make sure your ads maintain good click-through rates (over 0.5%) to avoid getting disabled. Run it for a week or two (preferably more) and you will have a good number of search referrals and “long tail keywords” that people are actually looking for. You can later create good content pages that include those keywords. In most cases, long tail keywords are really easy to rank with on-page optimization only.

I will probably write a Youmoz entry with more detailed instructions on how to take advantage of this. In this way I can get more people to try it and get really valuable feedback.

Here is the Python code:


# LongTailMiner.py v0.1 alpha by Hamlet Batista 2007


import re

from urlparse import urlparse

from cgi import parse_qsl

p = r'[^"]+"GETs([^s]+)[^"]+"s2[^"]+"([^"]+(?:google|yahoo|msn|ask)+[^"]+)"'

log = open('tripscan.actual_log')

lines = log.readlines()

keywords = set()

for line in lines:

 m = re.search(p, line)

	if m:

 	(internal, link) = m.groups()

		elements = urlparse(link)

		if elements[4]: #check to see if there is query string

 		params = parse_qsl(elements[4])#break qs in keyword, value pairs

			for (k,v) in params:

 			if k == 'p' or k == 'q':#top search engines use p or q for the keywords

 				keywords.add( elements[1] + " - " +  v)

#print the report

for k in  keywords:

 print k

Here is the output:

search.sympatico.msn.ca – best places to vacation in april
http://www.ask.com – help find a cheap vacation package anywhere
http://www.ask.com – new york vacation package deals
search.yahoo.com – vegas vacation packages
search.yahoo.com – what is the best beaches to stay in jamaica
search.yahoo.com – outrageous hawaii vacation packages
http://www.google.se – “paris in 5 days” versailles
search.sympatico.msn.ca – Vacation Package Deals
search.yahoo.com – vacationpackage
search.yahoo.com – vacation packages
search.msn.com – 10 best places for vacation
in.search.yahoo.com – vacation package
search.yahoo.com – best places to vacation in june/july
search.sympatico.msn.ca – best travel deals for june
search.msn.com – last minute caribean deals
search.yahoo.com – package vacation
http://www.google.com – Tripscan
search.sympatico.msn.ca – best places to vacation in June
search.sympatico.msn.ca – best places to travel in october
search.yahoo.com – vacation package
search.msn.com – caribean vacation
search.msn.com – Best Caribean vacation
http://www.ask.com – Cheap Vacation Package
search.sympatico.msn.ca – CANYON RANCH IN LENNOX
search.sympatico.msn.ca – find vacation packages
ca.search.yahoo.com – Hawaii all inclusive Vacation Packages
search.yahoo.com – california vacation ideas
search.yahoo.com – vacaton package
ie.search.msn.com – caribean vacation
search.yahoo.com – all inclusive package deals from New York to Cancun
search.yahoo.com – best places to explore
search.msn.com – caribean vacation island packages
search.yahoo.com – vancation package
search.yahoo.com – puerto vallarta nude resorts
http://www.ask.com – all inclusive vacation places
search.yahoo.com – vacation packge
search.yahoo.com – vacation package
http://www.ask.com – caribean deals
search.msn.com – best hotels in caribean
search.msn.com – the best caribean vacation
http://www.google.com – related:www.exectourtravel.com/
ca.search.yahoo.com – vacation packages

This is just scratching the surface. One improvement we can make, is to identify the landing pages to which the keywords lead, so we can make sure visitors are finding what they want.


In order to use the script you need to download Python from http://www.python.org. The script should run in Unix/Linux, Mac, and Windows but I only tested it in Linux.

1. Copy your log file to the directory were you saved the script.

2. Change the name of the log file (inside the quotes) in the line log = open(’tripscan.actual_log’) to the name of your log file.

3. In the command line type: python LongTailMiner.py and you should see the report.


2 Responses to “LongTailMiner v0.1 alpha – find long tail keywords nobody thought about”

  1. 1 Heather Paquinas June 4, 2007 at 7:59 pm

    Hittail essentially does this, but makes it much easier.

  2. 2 hamletb June 5, 2007 at 1:20 am

    Yes. You are absolutely right.

    I will try to make my tools accessible to the non-technical person.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

Subscribe to my Full RSS Feed

Add to Technorati Favorites

My Top Affiliate Sites

  • TripScan - "vacation package" - 50 million competitors - #13 - Yahoo
  • ReferralPay - "best affiliate programs" - 60 million competitors - #8 - Yahoo | 2 million competitors - #2 - Live
  • InkTrack - "ink cartridge" - 8 million competitors - #5 - Live
  • My Nice Gateaway


    %d bloggers like this: