Non-Indexable Pages - Ryte Product Insights (2023)

Without indexing, the affected URLs can no longer be ranked. Depends on the strength of each oneURLLack of indexing can lead to a huge loss of visibility. Therefore, you should regularly check your website for non-indexable pages. Ryte can help you with that!

misconfigured or "forgotten" noindex tags

You can use the noindex tag to exclude websites from search engine indexing. In this case, iHeadersunrise

inserted.

Crawlers like Googlebot can still follow links on the page, but don't index the URL. Normally it will be like thisno indexThe tag is used to temporarily prevent a URL from being indexed. This can be due to changes in the URL or changes in the URL structure, for example. However, if the tag is forgotten after the changes, the page will be permanently removed from the index. If still tagged without the "follow" suffix, followGoogle robotsthe links on the relevant page no longer. It also prevents other URLs on your site from being crawled and indexed.

How to find misconfigured noindex tags with Ryte Website Success:

In the Indexability menu, click the What can be indexed? section. You will then receive an overview of which URLs Google can index and where there are potential indexing problems.

Navigate to Nonidex using the robot bar in the diagram. After one click you will see all URLs tagged with "noindex":

Illustration 1:Identify URLs tagged "noindex" with Ryte Website Success.

Then what should you do?

• Remove any stray noindex tags from the URLs in question.

• Then let Ryte track your project again and check it accordingly.

Misconfigured canonical tags

With aCanonical-TagYou can point Google to the "Original URL" for URLs with the same content. Usually Google ignores this when indexingDuplicatey refers to the canonical URL only.

Many errors can occur when setting canonical tags. It is therefore possible that the canonical tag points to a page with a noindex tag. In this case, the canonical page is not indexed. Basically, the canonical tag is just a recommendation to Google, not an instruction. If Googlebot assumes that the canonicals are set incorrectly, they will be ignored. For this purpose, Google checks the links stored in the Sitemap.xml.

Although the canonical tag cannot theoretically be equated with the noindex tag function, in practice faulty canonical tags have similar effects.

What you should also note about canonical tags:

• Basically, the entire content of a URL is ignored if the stored canonical tag points to another URL and Google accepts it as such.

• For canonicalization to work, the pages in question must have very similar, if not identical, content.

  See Also

  .htaccess und SEO – Alles was Sie wissen müssen!Datenschutz • Die Stadt Überlingen▷ Top 10 Technical SEO Courses (Free & Paid)What should you consider when buying a website?

• If no canonical tags are placed in the

  -area they are ignored by Google.

• When a canonical tag points to a URL with a redirect, it takes Google longer to decide which URL to index.In this way you give away a valuable tracking budget. In such cases, it can also happen that Google ignores the canonical tag.

How to check canonical tags with Ryte Website Success:

In the Indexability section, go to Canonicals. From there, click Canonical Use. The overview shows you where canonical tags are used and where not on your website.

Figure 2:Verify canonical tag usage with Ryte Website Success.

Then what should you do?

• Include self-referencing canonical tags if you don't want a page to refer to a canonical URL.

• Check canonical tags and your link destinations. Everything is correct? Correct possible mistakes.

Fejl y robots.txt

the smallrobots.txtIt's just a text file stored at the root of your domain. But the file has a big impact on your website indexing. Because it contains instructions on how to allow bots to crawl your site. For example, you can use robots.txt to exclude bots or prevent all bots from crawling specific folders.

Robots.txt does not exclude URLs from indexing, which is often misunderstood. However, since the instructions prevent Googlebot from crawling the URLs, they cannot be indexed in this way either. By excluding folders, you can indirectly ensure that they are "not" indexed.

Because of this, you should check robots.txt regularly, especially after making changes to the file. In principle, it is advisable to let the Googlebot crawl as many areas as possible anyway. This ensures that Google can display your web pages as a human user would with their browser.

Common mistakes are:

• Exclude folders with files like CSS or JavaScript that are important for loading functions on the website.

• Exclusion of important categories.

• Using wrong syntax in robots.txt, which means whole folders cannot be accidentally crawled.

To manage your robots.txt file with Ryte:

In the Indexability section, click Robots.txt. You already get an overview of which URLs are excluded from file crawling.

Figure 3:Monitoring robots.txt with Ryte.

wrong redirects

redirects like this301 redirectHelp redirect old URLs to new URLs without losing link juice. Instructions stored in the Apache server's htaccess file are almost imperceptible to users. However, if the 301 redirect is configured incorrectly, there may be a loss in ranking, since the original page is no longer considered for indexing.

A common mistake is that a 301 redirect redirects to the wrong URL. It's also bad for indexing if a 301 redirects to a URL that points to another URL via Canonical, or redirects to a URL that is excluded from indexing via Noindex.

With Ryte you can control all redirects on your website:

In the "Indexability" area, click on the "Redirects" menu item and then on "Status Codes". The report immediately shows you all configured points.

Figure 4:Check the redirects against the success of the Ryte website.

You have to do this:

• Are 301 redirects still necessary? Check if the new URLs are already indexed by Google. You can do a simple "site query" with the correct URL. If the new URL already exists in the index, the 301 redirect can be removed.Google's Gary Illyes confirmed this in a tweet..

• Are temporary redirects (302) still necessary? These redirects are usually set up when there are new URLs to redirect due to seasonal or marketing reasons. When the actions are finished, you can also remove the redirect.

Broken or missing internal link

An intact inner joint functions as a network of veins that supply the body with fresh blood. If the internal link is broken, the distribution ofLinkjuiceAnd in the worst case, broken links prevent individual URLs from being indexed on your website.

Conversions or restructuring on your website can cause URLs to become so-called "orphan pages". These are websites with no internal inbound links. If these pages are not externally linked via backlinks, the Googlebot no longer has the opportunity to crawl and index them.

Ryte Website Success can help track down orphaned pages:

To do this, click on "Links" and then on "Pages without incoming links". The report then lists all orphaned URLs.

Figure 5:Find Orphaned Pages with Ryte Website Success.

So you should do the following:

Set the internal link again to the appropriate URL or folder. Correct the relevant link in the menu or content.

wrong pagination

URL pagination combined with unindexed and canonical tags can annoy search engines and therefore result in incorrect indexing or no indexing at all.

For example, if you have inserted a pagination and the paginated pages refer to the first page of the pagination, the paginated pages may no longer be indexed. Thepaginationbut to show Google that it's a continuous series of pages.

Pagination with the noindex tag can also often have potential errors. For example, it would be a mistake to exclude a paginated page from indexing with "noindex" and have that page refer to the first page of the pagination with a canonical tag.

In this case, Google is instructed not to include the page in question.IndexThat should be the case, but it would tell the bot to only consider links on the first page of pagination. This would interrupt the harmonious "circulation" of your page: the link juice could no longer reach all category pages.

How to find the errors with Ryte Website Success:

In the Indexability section, click Pagination, and then click Pagination Target.

Ryte will then show you where the links marked with rel="prev" lead to.

Figure 6:Check pagination with Ryte Website Success.

You have to do this:

• Make sure the page reference is correct and the highlighted links point to the correct page or folder. Correct any errors.

• After the improvements, let Ryte recrawl your project and check the pagination again.

Pages with 404 status code

A 404 error page is displayed whenever the server can no longer find content for the requested URL (404 not found) in response to a client request. Now, if Googlebot crawls your website and encounters error pages frequently, the crawling will be severely disrupted. This can lead to the fact that the Googlebot can no longer reach the deeper areas of your website due to the limited crawl budget. There is such a high risk that these URLs will not be indexed due to lack of crawling.

How to find 404 errors with Ryte:

In the Indexability section, go to Status Codes. Then click the red ">4xx (not found)" button. The report will then show you all the URLs that are returning a 404 error code.

Figure 7:Get status codes with Ryte.

You can do it:

• Check that there is actually no relevant URL for the 404 errors that are present. If not, add a 301 redirect for new content.

• Check if the URLs for which the server returns a 404 error are incorrect.

• Check if you're using a custom 404 error page that offers users possible alternatives.Google also recommends using custom 404 pages. Because 404 errors cannot always be avoided and are a "natural" part of the web.With a well-designed error pageHowever, it can increase user-friendliness and prevent many users from churn.

Diploma

The larger a web project is, the more likely it is that there will be errors. This makes it all the more important that you regularly check your project for errors, especially on websites with many URLs. It's always a good idea to check if you're making changes to the content, URL structure, or redirects. With Ryte, you can generate relevant reports to help you troubleshoot with little effort.