Log Analysis

Demo Data nginx_combined

demo_access.log.gz · 2026-05-29 → 2026-06-04 · Week-over-Week comparison

Total Bot Requests

80656

↑ 2.6% vs prev period

HTML Pages Crawled

30368

↑ 1.6% vs prev period

Active Bot Types

across all crawlers

Crawl Errors (4xx+5xx)

8784

↑ 2.6% vs prev period

Crawl Volume Timeline

Daily bot requests — current vs previous period

Current Previous

By request count — current period

Desktop · Mobile · Image · Ads · News · Inspection

What bots are actually fetching

2xx / 3xx / 4xx / 5xx — current vs previous

Identify crawl windows and budget peaks

Key metrics across periods

Metric	Current	Previous	Change

Forward-confirmed reverse DNS — detect impostor bots

Verified Unverified

IP Address	Bot Name	Reverse DNS	Requests	Status
66.249.64.2	Googlebot Mobile	crawl-66-249-64-2.googlebot.com	14798	Verified
198.50.128.1	MJ12bot	crawl-198-50-128-1.mj12.co.uk	14521	Verified
149.56.123.1	SemrushBot	crawl-149-56-123-1.semrush.com	14243	Verified
66.249.64.1	Googlebot Desktop	crawl-66-249-64-1.googlebot.com	13743	Verified
66.249.73.2	Google Inspection Tool	crawl-66-249-73-2.googlebot.com	13705	Verified
20.191.45.1	DuckDuckBot	crawl-20-191-45-1.duckduckgo.com	12695	Verified
185.191.171.1	Unknown Bot	unknown-185.191.171.1	10416	Unverified
203.208.60.1	Unknown Bot	unknown-203.208.60.1	10301	Unverified
40.77.167.1	Bingbot	crawl-40-77-167-1.search.msn.com	9118	Verified
66.249.64.3	Googlebot Image	crawl-66-249-64-3.googlebot.com	7598	Verified
66.249.73.1	AdsBot Google	crawl-66-249-73-1.googlebot.com	6368	Verified
17.58.98.1	Applebot	crawl-17-58-98-1.applebot.apple.com	5691	Verified
180.76.15.1	Baiduspider	crawl-180-76-15-1.baidu.com	5434	Verified
54.36.148.1	AhrefsBot	crawl-54-36-148-1.ahrefs.com	5177	Verified
5.255.231.1	YandexBot	crawl-5-255-231-1.yandex.ru	842	Verified

Most frequently requested HTML resources

#	Path	Hits
1	/	13644
2	/blog/	7769
3	/products/	6398
4	/blog/how-to-improve-seo/	5746
5	/blog/technical-seo-guide/	4975
6	/blog/keyword-research-tips/	4010
7	/pricing/	3802
8	/products/enterprise/	3516
9	/about/	3383
10	/blog/link-building-strategies/	3180
11	/products/pro/	2769
12	/contact/	2707
13	/blog/core-web-vitals/	2691
14	/products/starter/	2687
15	/blog/e-e-a-t-guide/	2123
16	/blog/structured-data/	2118
17	/docs/api/	2018
18	/docs/getting-started/	1952
19	/products/compare/	1952
20	/robots.txt	1434