SEO Crawability

ben.wright.dev · 23 July 2020 10:33

Hey all,

My website had decent google presence while using PHP wordpress install. However, after moving to Frontity (using head-tags), I have noticed it has been removed from google search results.

I have now manually set the robots: https://benwrightdev.com/robots.txt

But this checker still says there is a problem with crawability?
https://freetools.seobility.net/en/seocheck/benwrightdev.com

I’m not missing anything right? That robots.txt should allow googles crawler?

Any help greatly appreciated,
Thanks,
Ben

ben.wright.dev · 24 July 2020 08:36

Ok I think I solved this. I had enabled ‘Discourage search engines from indexing this site’ on my wordpress install and because frontity/head-tags takes the whole head section, I had a no-index meta tag on my frontity installation.

I have disabled this option and purged the wordpress head tags plugin cache and it looks good.

mburridge · 24 July 2020 08:56

Hi @ben.wright.dev

Awesome, glad you managed to solve your issue. It’s alway worth checking your WordPress configuration and not just assume that it’s a problem with your code. I’ve fallen foul of that one too!