Manage your crawl budget effectively

Bastian says: “I’m going to talk about how to tackle the growing challenge of managing crawl efficiency and crawl budget in an era where search engines like Google are overwhelmed by the sheer volume of content that’s currently being produced.”

How do you manage Google’s capacity to crawl what you have to offer?

“Right now, we have this content explosion. Everyone and anyone is using AI-supported tools that make content creation faster and cheaper than ever – and it’s creating real challenges for Google. They’re struggling to crawl and index it all. Despite all the fancy algorithms they have, they just can’t keep pace with this increasing volume. That is the biggest problem.

Managing crawl budget is really crucial, and it involves best practices that we have talked about for a long time. It’s not rocket science but it’s even more important this year. Years ago, you would only have done this extensively with sites that were very sizable. Right now, even a mid-size site with 5/6 figures of URLs will see benefits.

To go about it, there are a handful of things that are straightforward, but it’s worth quickly echoing them. A good starting point is the consolidation of low-value pages. Merge or eliminate the pages that are either thin, duplicate, or don’t bring any value. Deleting URLs/inventory will ensure that other stuff is crawled more frequently.

The same goes for internal linking. That’s a topic that SEOs talk about a lot, and rightfully so, but it does make a big difference if you link the right pages with each other and build a certain hierarchy. It helps Google to crawl and index the most important pages on a domain or site.

From a tooling perspective, you can use robots.txt, meta tags, and the X-Robots-Tag which prevent crawling and/or indexing. Depending on the use case, this is the weaponry that you have in your arsenal.

Then, there are XML sitemaps. Again, this is not brand new, but a lot of sites struggle with maintaining or creating valid error-free site maps in the first place. It’s a strong signal for Google – also from a canonicalization standpoint – of which URLs are more important than others.

There’s also a protocol called IndexNow. Even though Google does not officially support it, Bing does, Yandex does, and others do as well. That will help with overall efficiency for other engines as well.

That is how I would go about it. The other part of this is how you can combine that with the hip topic of AI. We’ve talked a bit before about AI tooling on the content side, but there are also a whole lot of technical tasks like automating internal linking, redirect management, and URL mapping that you can do with the help of AI tools.”

Is it more about the quality of pages than the total number, and being honest with yourself about which pages Google would want to have in their index?

“Depending on things like how trusted your domain is and how old it is, you’re allowed more or fewer URLs in Google’s index. Generally speaking, though, you should look at the pages that actually provide value to users.

Information gain is incredibly important. There might be pages where there’s no information gain, like your privacy policy or a contact form. They don’t necessarily drive organic traffic so they are pages that might not necessarily need to go into the index. If you have a navigation set-up, certain types of categories don’t provide any value. These types of things.

Understand your inventory, and then ask yourself – for types of pages or types of templates – does it make sense that people would end up on those pages when they come in from Google? That is the approach. In that way, less is more.”

For healthy growth, is there a maximum percentage increase to your existing web pages that you would recommend per month?

“Considering the relation to your current content is a solid way of approaching it. If you have 100 pages and you start adding thousands more every month, that’s a spike that would not look natural, for many reasons. That might catch Google’s attention. You might pass a certain threshold, and then you’d be in trouble.

The bigger question is not so much about the number of pages but, for what you do create (whether it’s AI-assisted or human-created), does it actually provide value to the audience that you’re trying to attract? If that’s the case, I think you’re okay. If not, then that is the bigger problem.

It’s not so much about adding 10, 11, or 12 pages, but whether those pages bring value. Do they fit in contextually and from a semantics perspective? Do they provide value to the users that would end up on your site? I don’t think it’s necessarily about whether it’s AI or not.”

What does quality AI-assisted content look like?

“I don’t think it looks much different from content that’s purely human-made. If you look at it from a process standpoint, what querying a large language model does really well is shortcut parts of the research process. Obviously, you still need to fact-check, but it can be a shortcut with that.

Once you have done the research, it can give you a shortcut to turn the whole thing into a document outline, for example. It can even help you pre-draft certain sections of that content.

The major difference is between content that is purely AI versus AI-assisted. Assisted means that all the facts have been human-checked and quality-assured because there’s always stuff in there that’s definitely wrong. No matter how well you prompt, even if you are using a tone of voice and providing input samples, there will always be giveaways in the grammar and language it has come from a large language model. You need the human editing component as well.

The underlying problem remains the same, whether it’s done by a human or an AI, so it doesn’t look different in terms of output. If it doesn’t provide value, because it’s commodity content that’s already on the web, then it’s not worthwhile. ‘10 Travel Tips for Greece’ is something that Google can consolidate based on all the sources they have already vetted. If you go and curate that, whether you use AI or not, there’s no real information gain.

However, if you can add a very unique perspective on a certain destination in Greece, because you’re the only person talking about it who has been there recently, that’s an information gain. That’s the difference; does it actually provide value? It sounds abstract, but it’s a simple question.

Also, when you assess these things – when you’re doing consolidation, cleanups, and merging things – you need to be very thorough and very honest with yourself. There will be bits and pieces that are older, outdated, and not relevant anymore. They don’t provide value, so they can go.”

Is there a particular AI tool that you recommend for content creation at the moment, and is there a process that works best?

“From a tooling standpoint, it depends on language, but also style. None of the large language models can be used out of the box right now. That would just return standardised garbage. If you are more in the ChatGPT world, you want to use a custom GPT where you can pre-train it with the context and an example of what you expect it to produce. Provide it with input, but also example output for format, tone of voice, etc.

Custom GPTs are pretty cool for that because, when you share it between team members, they don’t have to redo these things a million times – where someone will do it differently and all of a sudden you would have a different output.

I do like to work with Claude quite a lot. They do a really good job. Oftentimes, even in the draft stage, it already sounds much more natural. There are a whole bunch of more specialist writing tools as well, so it really depends on the type of content you want to produce. If you start scaling, then you’re moving away from an interface anyway. You’re more on the API side.

We’ve been building a lot of stuff around the OpenAI API. You have things like prompt sequencing, and you have many more settings that you can play with to really come to the output that you want or need.

If you are using it to help with internal linking, then training your own model with your own content is probably the way to go – especially at enterprise scale. Default matching one piece of content to another is nice, but it’s not integrated enough. You will obviously produce new content, so how is that being interlinked into this whole thing?

For one of our clients, we created an integration with large language models. We dump in newly produced content and refreshed content as well. Then you have all the semantics in the model, and it can give you back the best possible link targets, one way or the other. It’s a living process, rather than a static one. We’re not just doing internal linking once and then it’s gone. Integrating it on the CMS side can make a big difference.”

If an SEO is struggling for time, what should they stop doing right now so they can spend more time doing what you suggest in 2025?

“A lot of the tasks that have historically been done manually or semi-manually, in Excel with different mini tools, can be now pre-produced in a large language model. Question your processes and your deliverables.

I’m not saying to stop one or the other. I’m saying you should try to squeeze for more efficiency in existing things. It depends on what your job is, but if you have been doing schema markup, meta descriptions, or page titles by hand, that is something that you can draft at scale.

The same is true for redirects. If you’re doing a migration and you have to map out the old URLs to new URLs, you will get 80-90% accuracy by letting the machine do the work, and then you check it. It’s much faster than starting from scratch and doing it all manually.”

Bastian Grimm is CEO at Peak Ace AG, and you can find him over at PeakAce.agency.