Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Moz crawling bot
-
Hi guys,
in OpenSiteExplorer -> Top Pages, there are no page titles displayed in a raport for certain domain, and "HTTP Status" column shows: "Blocked by robots.txt".
I tried to find out what the ID of Moz crawling bot is, and on this page: http://cloudz.click/community/q/seomoz-spider-bot-details someone says it's: Mozilla/5.0 (compatible; rogerBot/1.0; http://www.seomoz.org/dp/rogerbot).
However, my robots.txt doesn't have such entry. Take a look:
Automatically banned scanners and crawlers section
User-agent: 008
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: metajobbot
Disallow: /
User-agent: Exabot
Disallow: /
User-agent: Ezooms
Disallow: /
User-agent: fyberspider
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: MojeekBot
Disallow: /
Section end
What could be the problem here, then? Why does the Moz bot think I'm blocking it?
-
Thank you Keri, problem solved.
-
If you pull out dotbot from your exclusions, you should be fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz link explorer is how much reliable and accurate as compared to others link explorer tools like Ahref
Hi.. I need to check backlinks for one of my client website duplicatebills through Moz Pro link explorer tool then its show that linking domain to this website are 90 but when i cross check these links from other tools like Ahref free backlinking tool then its show a different figure (ahref show that linking domain to this website are 103).
Link Explorer | | dbills
I am confused that which one correct. And MOZ link explorer tools in reliable to which extent?0 -
Ranking Keywords are not showing on Moz
i just check my Apk Finds on moz site explorer and check its ranking keywords but it does not show it please help me my site age is 3 months
Link Explorer | | malik12378670 -
How is Moz DA affected by spam links? Disavow file?
So it does not appear that moz let's you upload your disavow file. So when moz calculates your DA how do spammy links factor in? After digging through our GA it appears our site was hit with the 2016 penguin update and never recovered. Our weekly visitors were 2k, then dropped to 500 and have stayed close to that level for a while. We've used the disavow tool, without success over the past 3 years. During that time we have done link out reach and built around 10 legit good quality DA links since. But we have not recovered. At this point i'm thinking I should just remove the disavow file. Moz says our spam score for our domain is 5%.
Link Explorer | | jessicapremier0 -
Moz tool to estimate a website's organic traffic?
Moz has tons of great tools, so I was quite suprised there is nothing similar to Alexa's siteinfo tool, or SimilarWeb's site comparison tool. Am I missing something? Or does such a tool exists at Moz? Or is there a different way of estimating my competitor's traffic? It would be great if Moz would have something like that.
Link Explorer | | GrandRapids0 -
If I have a MOZ PRO account, do I still need Screaming Frog?
Hi, I have a MOZ PRO account which is brilliant for me. I am about to move my site from HTTP to HTTPS and was looking for a smooth way to list every page and its details so I can make sure I do the correct redirects etc.. before the switch to SSL. Do I need to get Screaming Frog, or can the tools with MOZ Pro ease my path just as well? Thanks
Link Explorer | | SeoSheikh0 -
Fastest way to get new backlinks recognised by Moz?
Hi all, I have seen a few discussions / advice pieces around this, but wondered what the current view is on the fastest way to get new backlinks recognised by Moz? I have seen people say that the content needs to be shared (e.g. on Twitter), and advice around creating Bitly links. I am not convinced that these techniques are working. On a related note, my website is very large. I have around 2.5m pages. Most of these pages are 3-4 clicks from the home page, but I have a small amount of important content much higher up in the site (1-2 clicks from the home page). Does Moz crawl pages nearer the home page more frequently, or does it make a pass of all pages on the site before it starts at the beginning again? If it makes a complete pass of all pages, then presumably making my site smaller will help to get Moz to see new content more quickly? Is there any other value to be gained in making a large site smaller? Many thanks in advance, Neil.
Link Explorer | | NHARRIS10 -
How to force moz to crawl my backlinks?
I have some good number number of backlinks in my webmaster tools. But, open site explorer is showing very few backlinks. How to force moz to crawl all the backlinks? Or is there any way to submit backlinks to moz?
Link Explorer | | sankar7890 -
301 Redirects: How long for Google to recognize? How long for Moz/OSE to recognize?
I redirected a few pages that had funky URL issues (capitals and underscores and useless words) to the same page renamed with proper URLs. The sitemaps were changed also, and re-submitted. Like this: mydomain.com/Not_Pertinent_Words.html >>> mydomain.com/good-words.html Google seems to have found them and changed the search engine results listing in about 8 days. But it's been about a month now and Moz and OSE still have not transferred all the strength and link data from the old URLs to the new ones. Question 1: How long does it usually take Google to transfer all the link and strength data for a 301? Question 2: How long does it take Moz and OSE to do the same? Is there something I need to do to tell them about the changes?
Link Explorer | | GregB1230