You may have already encountered SEO dashboards, such as those from Screaming Frog, that sync with Looker Studio. Maybe you use an enterprise tool to track your technical SEO in the cloud.
This is all good, but how useful is it?
Does this improve your ranking? If you manage 30-50, or even 100 or more sites, does tracking technical SEO with cloud crawlers offer anything actionable?
If you said yes, then great! I’m happy for you.
But if, like all of us, you wish there was a better solution, there is!
The key is to be more targeted and intentional about which pages we follow.
This article explains what is essential to follow and how to follow each of them. I’ll also cover useful paid tools for tracking SEO elements. (I am not affiliated with these companies, but I am a customer of a few.)
Indexability elements
Monitoring the indexability of your site is crucial. This should be at the top of your list. If your site is not indexable, all your efforts are in vain.
In this section, I’ll explain the main indexability factors that affect your visibility on Google and how to track them.
1. Changes to the Robots.txt file
THE robots.txt file is the first file that search engines look at when crawling your website. This simple file provides instructions on how bots should crawl a website.
What are we following?
Every SEO should keep track of changes to the robots.txt file, especially if it starts blocking search engines.
Many tools offer this feature, but it’s important to set up email alerts to notify you if the file blocks search engines.
A common reason for this is when developers move a site from staging to production and accidentally transfer the robots.txt file.
This happens when all files are uploaded instead of just the updated ones.
What tools to use
While I am inclined to build custom Python tools for tasks, why reinvent the wheel here? Several inexpensive tools work great for validating your robots.txt file.
I prefer LittleWarden because it lets you track specific changes or general indexability checks and send email alerts. You can set it to check daily or hourly.
However, Visualping (which I used in this knowledge graph case study) is also a great choice for tracking changes to the robots.txt file.
2. Noindex Bot Tags
Ah, the famous tag without index. This is a robots meta tag that you can add to pages you don’t want indexed in Google, like login, account, or other low-value pages.
Whether you absolutely need to index pages or not, tracking changes to your setup is essential to your SEO health.
What tools to use
LittleWarden is great for this because you can customize the settings for each page, allowing you to easily set most pages to be indexable while marking a few as unindexed.
Tools like Screaming Frog and Sitebulb can also work, but they require more setup.
3. Changes to the X-Robots-Tag
In simple terms, the x-robots-tag does the same thing as the noindex meta tag, but instead of being in the
of your website, it appears in the HTTP response headers.
What tools to use
LittleWarden comes pre-configured to check your HTTP response headers for any indexability issues that may arise, including canonical tag changes.
4. Validation of the XML sitemap
XML sitemaps are another powerful SEO tool. These files contain a map of all the links present on our sites.
Google uses these as useful clues for which pages they should discover and add to their crawl queue.
If your sitemap has errors (e.g. failed to fetch or crawl), Google will keep trying to treat it for a few days. If attempts persistently fail, Google will stop trying to crawl the URL.
What tools to use
LittleWarden wins again with its ease of use for tracking changes to your XML sitemaps and ensuring their validity.
5. Canonical Tag Changes
Canonical tags are an element often misunderstood and misused in SEO.
Although Google only treats them as indices and not strict directives (like noindex), it is still important to know if they have been changed on a page.
What tools to use
LittleWarden can track this, but you can also use tools like ChangeTower or VisualPing. Screaming Frog and Sitebulb can also track it, but setting up email alerts requires additional steps.
On-page ranking elements
On-page SEO Elements refer to factors on your pages that users and search engines can see.
If you are in local SEOyou may not encounter many changes on the site. However, when you add other members of your team with access to the site, someone may inadvertently make marketing changes that could affect your rankings.
This is why tracking elements on the page is crucial.
6. Title Tag Changes
Title tags are any text formatted using a formal HTML title (
Google representatives say they don’t matter much for rankingsbut they do it for accessibility.
Many SEOs, including myself, believe that a well-optimized H1 and heading structure can improve rankings, but that’s a discussion for another time.
For now, it’s important to track changes to your H1 tags on your most important pages.
What tools to use
This is where the variety of tools we can use really opens up. You can test any of the following below with reliable consistency:
Little guard. Versionist. Wachete. Visualization. Change the tower.
Again, you can technically use Screaming Frog and Sitebulb, but setting up email alerts from that will be a pain.
7. Changes to internal links
Track changes to your internal links can be essential to maintaining a strong internal linking graph.
If a well-placed internal link with optimized anchor text is removed or changed, it could affect your rankings.
Will this affect him much? Probably not. But are you really ready to test this for us?
What tools to use
Little guard. Versionist. Wachete. Visualization. Change the tower.
8. Keyword Changes
Tracking changes to your keyword usage on your site is essential to your SEO success.
If another marketer on your team tries to improve your content, they could unknowingly improve your page ranking.
What tools to use
Little guard. Versionist. Wachete. Visualization. Change the tower.
SERP Visibility
Tracking your appearance in the SERP may not directly affect your rankings, but it could affect your organic click-through rates (CTR).
9. Title Tag Changes
Title tags are the code we provide to search engines that suggests what we think the displayed title should be.
Yes, Google changes title tagsbut that doesn’t mean we should ignore them.
What tools to use
Little guard. Versionist. Wachete. Visualization. Change the tower.
10. Meta Description Changes
Meta descriptions is perhaps one of the most controversial topics in SEO. Some things that make this topic so interesting include, but are not limited to:
Technically, these are not ranking factors, but they are essential to our sales pitch in the SERP. SEO tools warn us about pixel and character length, but Google will change them anyway to match search intent. I directly heard Gary Illyes at a conference tell an SEO specialist that optimizing title tags and meta descriptions is a waste of time. (I take umbrage.)
However, if you’re already tracking unintentional changes to your site, clicking on the meta description box is worth the extra second.
What tools to use
Little Warden Versionista Wachete Visualizing ChangeTower
11. Schema validation
Schema markup is a powerful tool. It can send important structured data to Google and help us create interesting SERP features for our results.
Creating and implementing schema markup takes time, so it’s important to track any changes, especially since SERP features can increase your organic CTR.
What tools to use
Little guard. Versionist. Wachete. Visualization. Change the tower.
Good follow-up
Now that you have a comprehensive list of SEO elements to track, it’s time to add robust monitoring to your SEO campaigns.
Trust me, it’s better to be prepared than to discover a change a few days later.
Contributing authors are invited to create content for MarTech and are chosen for their expertise and contribution to the martech community. Our contributors work under the supervision of the writing and contributions are checked for quality and relevance to our readers. The opinions they express are their own.
