Responsive Site Failed Google Mobile-Friendly Test
Responsive Site Failed Google Mobile-Friendly Test
Ever had this issue where your responsive site failed Google Mobile-friendly Test? It’s most likely caused by the site’s robots.txt file.
There are lots of articles about this but my situation is unique so hopefully you could learn from this. So after I ran my site through GMFT page and discovering that my site indeed failed this test, I researched the issue and opened my robots.txt file.
I found the line below and then removed it from the robots.txt file:
Disallow: /wp-content/
Then I ran the test and it showed that my site passed the test with flying colors. Fast forward to two days later, disappointingly my site failed again. It should have been quite obvious to me that this line in my robots.txt is the last piece of the puzzle.
Disallow: /wp-includes/
So I removed this line above and it finally passed the mobile friendly test. Obviously removing these two lines from robots.txt addressed the Mobile-Friendly test because those 2 lines were stopping the Google Bots from accessing my stylesheet and javascript files.
That still didn’t explain why the Google Mobile-Friendly Test (GMFT) page did not provide consistent results, which makes it super annoying.
To make the issue confusing, there are lots of articles that recommend you disallow Google bots to crawl the “wp-content” subfolder. Also the “wp-content/plugins/” directory used to be a recommended disallowed line item, but according to a prominent SEO expert (and others), this would be a self-sabotaging practice. This is because the wp-content/plugins might have Javascript and CSS styling that Google needs to access to render your page properly.
So what is the best content for the average WordPress site Robots.txt?
Well according to Yoast, don’t block anything, not even your wp-admin page. I find his explanation is lacking on this and missing the point though. why even have a WordPress login page that has the “wp-admin” in your URL? I’m all for making it hard for hackers to find my login page even if it’s not foolproof. See Yoast’s blog post regarding this topic HERE.
The takeaway for me on what is the best practice for your robots.txt is just don’t even touch it. The exception is if you have specific secret pages you truly don’t want to share even if it has some juicy links inside them, such as download page for paying customers.
