Log file analysis for SEO: what it is and how to do it

SEO log file analysis means reviewing your server's records to see which URLs Googlebot (and every other bot) actually requested, when it requested them and what your site answered. It sounds like an IT task, and partly it is, but for SEO it is the only way to know what happened on your site without relying on what a tool summarizes.

A made-up example, so the numbers are visible. An ecommerce site gets 800,000 Googlebot visits a month. That sounds like great news, until someone opens the logs and finds that 300,000 of those visits went to color and size filters nobody wants to see in Google. Meanwhile, the new collection launched ten days ago has not received a single visit.

My take is that logs should be read by template and by bot type, because that is how you see which part of the crawl reaches the pages that sell and which part gets lost. Here I cover what a log file contains, where to get it, how to run the analysis step by step and what to look at, including the AI bots that visit your site today without showing up in any dashboard.

What is SEO log file analysis?

Every time someone requests a page, the server writes a line in a file: who asked, which URL, when and what it answered. That is an access log. It writes it for people, for Googlebot, for ChatGPT and for the scraper someone left running at home.

SEO log file analysis takes that file, filters the search engine visits and answers concrete questions: which sections does Google crawl and which does it ignore? How often does it get an error? How long does it take to reach a new page? It is a classic part of the technical audit of a large site, because it works with facts.

On a 300-URL site this rarely changes your technical SEO plan. On a 200,000-URL site, with filters and internal search, server logs usually set the roadmap for the next few months.

What does a log file contain?

A typical line from an Apache or Nginx server (in the combined format almost everyone uses) looks like this:

66.249.66.1 - - [10/Nov/2026:08:14:22 -0300] "GET /sneakers/running/ HTTP/1.1" 200 18342 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

It looks like hieroglyphics, but it has only a few pieces:

For SEO log file analysis, the fields that do the most work are the IP, the URL, the status code, the date and the user-agent. There is one that almost never comes by default and is worth asking for: response time (in Nginx it is $request_time; in Apache, %D). Without it you cannot see which URLs respond slowly to Google.

How is it different from Search Console and a crawler?

Fair question, because Search Console already has a Crawl Stats report (under Settings) covering the last 90 days. And a crawler like Screaming Frog goes through your site as if it were Google. All three help with SEO, but they answer different things:

For an audit I would always start with Search Console, because it is free and already there. If you see something odd (error spikes, lots of requests to URLs you did not expect), logs are the next step to find where the problem is.

Where do you get your server logs?

Here comes the least glamorous part of SEO: you almost always have to ask someone for the files, and where they live depends on how your site is set up.

• Your own server (Apache or Nginx). They usually live in a folder like /var/log/ and someone in IT can export them compressed.

• Managed hosting. Some show them in the dashboard; with others you have to ask support.

• Behind a CDN. If you use Cloudflare or another CDN, many visits are served from cache and never reach your origin server. In that case you need the CDN logs as well as the origin logs (and not every plan delivers them in full).

When asking for them, I would put three things in writing: at least 30 days, every subdomain and the fields I mentioned above, unfiltered. If the IT team worries about user IPs (rightly so, they are personal data), you can ask for bot lines only or for anonymized human IPs: for SEO that is enough.

How do you do log file analysis step by step?

This is the method we use in the technical SEO audit of a large site. On a small site you can do the same with a spreadsheet.

1. Collect 30 days of logs. With less, one odd week (a launch, an outage) can fool you.

2. Verify the bots. Keep only the visits that really come from Googlebot (I explain how below) and put AI bots in their own group.

3. Group URLs by template. Product pages, categories, filters, parameters, pagination, internal search, articles. This is the step that pays off most, because 40,000 product pages grouped together show patterns that get lost URL by URL.

4. Compare with what should happen. Compare the URLs the bot visited with your sitemap and with a crawl of your site. The differences are the diagnosis.

5. Prioritize by what sells. Rank findings by how much crawling is lost and by how much they affect commercial pages.

6. Measure again. After every change, pull fresh logs two to four weeks later and compare the distribution by template.

Step 3 is the key, because that is how log file analysis shows you what happened, which is not always what the sitemap says should happen. If you are interested in crawl budget, in the crawl budget article I explain when that crawling deserves a project of its own.

How do you know a visit really comes from Googlebot?

Anyone can write "Googlebot" in a script's user-agent, so that field alone proves nothing. Google documents two ways to verify it:

1. Reverse and forward DNS. To verify, you run a reverse DNS lookup on the IP from the log, check that the name ends in googlebot.com, google.com or googleusercontent.com, and then run a forward lookup on that name to confirm it returns the same IP.

2. Published IP ranges. Google publishes JSON files with the ranges its crawlers use (for example, common-crawlers.json for Googlebot) and you can compare the IPs against that list.

Log tools run this verification automatically. A quick hint (this comes from experience; Google does not document it): if your logs show far more "Googlebot" than what appears in Search Console's Crawl Stats, a good share of those visits are probably impostors.

What should you look for in the logs?

With bots verified and URLs grouped, I would look at four things, in this order.

Where does the crawl go?

Add up Googlebot visits by template and work out the share each one takes. It is the number that helps log file analysis the most, because it shows how often the bot comes back to each type of page. On sites with faceted navigation the pattern repeats a lot: filters and parameter combinations (color, size, sort by price) generate thousands of URLs, and Google keeps discovering and visiting them.

Back to the made-up example. If those 300,000 visits (38% of the crawl) go to filters with noindex or with a canonical to the category, Google is crawling plenty; the problem is what you are letting it crawl. Those are template decisions: which filters generate linkable URLs, which get blocked in robots.txt and which get consolidated.

Which status codes does Googlebot get?

Group responses by code. The healthy picture is that the vast majority are 200 (page OK) or 304 (not modified since the last visit). Some guides give thresholds, like under 10% redirects and under 5% 404s; they work as a reference from whoever publishes them (Google gives no thresholds). What I would look for:

• Lots of 301 redirects. They usually come from links pointing to old URLs or from chains left over from a migration.

• Lots of 404s. Deleted categories or products, never redirected, that Google keeps visiting (more on this in the 404 error guide I wrote for Ahrefs in Spanish).

• 5xx errors at specific times. A 500 error or a 503 error that shows up right when Googlebot comes through in a burst may not show in your monitoring, because monitoring checks once a minute and the bot requests dozens of pages in a row.

Which important pages does it not visit?

This is my favorite. Take the list of pages that matter to the business (or your full sitemap) and compare it with the URLs Googlebot visited in those 30 days. The ones that do not appear even once are, for practical purposes, orphans from Google's point of view.

And for every new page that sells, measure how many days passed between publishing it and Googlebot's first visit. If a new product page takes weeks while a filter gets visited every day, you have a problem measured in something the business cares about. The fix is often to bring it closer to the home page with internal links from pages the bot visits often (in how to index a page on Google there is more detail on what to check next).

How fast does the server respond?

If you asked for response time, sort URLs from slowest to fastest. Google adjusts its crawl rate to what your server can handle: if it responds slowly or with errors, it slows down. A template that consistently takes more than a second to respond (usually listings with lots of filters) deserves a conversation with development.

Which AI bots show up in the logs?

This is the new part of technical SEO, and the reason I would ask for logs today even on mid-sized sites. AI bots do not show up in Search Console or Google Analytics, because they do not run the tracking code. Where they do show up is in the server logs, on the same lines where Googlebot appears.

OpenAI, for example, documents its bots and each one does something different:

The third row is the one I care about most. ChatGPT-User visits are the closest thing you have to seeing which of your pages ChatGPT reads to answer someone, so it is worth checking closely which URLs they land on. Other companies have their own (ClaudeBot, PerplexityBot), and it makes sense to group them the same way: training, search and user.

A practical detail: some CDNs and firewalls already block AI bots by default. If your logs show not a single OAI-SearchBot visit, check the configuration before concluding that ChatGPT is not interested in your site. To decide what to let through, in robots.txt and AI training I explain the new signal, and in ChatGPT traffic in Google Analytics how to measure the human visits that come afterwards.

Which tools work for log analysis?

Tools are chosen by volume, because the logic of log file analysis is the same for all of them:

• Spreadsheet. For a few thousand lines, you filter by user-agent and build a pivot table by folder and status code.

• Screaming Frog Log File Analyser. It reads Apache, Nginx and IIS logs, verifies bots and shows you the most and least crawled URLs. It connects with Screaming Frog SEO Spider, so you see what is crawlable and what was crawled on one screen.

• Technical SEO tools for enterprise sites. Oncrawl, Botify, JetOctopus or Seolyzer ingest logs continuously and cross them with crawls and Search Console. They make sense when the site has millions of URLs and Googlebot comes by daily.

• BigQuery or an ELK stack. When the data team already works there and the log files weigh several GB a day.

To start, Screaming Frog with 30 days of logs is enough for many of the sites we see.

How often is log file analysis worth doing?

It depends on the size of the site and how much it changes:

• Sites with more than 10,000 URLs or a catalog that changes often: once a quarter, plus after every big template change.

• Before and after a migration: always. It is the most direct way to check that Google found the new URLs and that the redirects work.

• Small, stable sites: once a year, or when crawl frequency changes suddenly in Crawl Stats.

If your site has few pages and Google indexes them without delay, log file analysis will not be your highest-impact SEO project, and it is fine to spend that time on something else.

What I would check this week

If you have never looked at your logs, I would do only this: ask IT for 30 days of server logs (or CDN logs), keep verified Googlebot and add up visits by page type. With that table you will know whether the crawl goes to the pages that sell or to the filters, which is what matters most for your SEO.

While you are at it, search for "ChatGPT-User" and "OAI-SearchBot" in the same file. If they do not show up, you already have a concrete question for the team that runs the CDN.