Patrick Hathaway, co-founder of Sitebulb, shares how the tool evolved into a leading technical SEO platform by focusing on user-centric features like hint-based reporting and JavaScript rendering. He also explores the future of SEO in an AI-driven world and the importance of staying grounded in fundamentals.
In this episode of the Page 2 Podcast, we sit down with Patrick Hathaway, co-founder and CEO of Sitebulb, a standout technical SEO crawler tool that has evolved from a desktop solution to a powerful cloud platform. Jon Clark and Joe dive into how Sitebulb carved a niche in a crowded SEO tool landscape, what makes its "hint-based" reporting so game-changing, and how it's built to thrive in the AI and JavaScript-heavy SEO future.
Patrick shares the behind-the-scenes of Sitebulb's product strategy, the role of customer feedback in tool development, and why structured data and fundamentals still matter. Learn about the challenges of SEO in the AI era, log file analysis for enterprise, and new insights into how LLM crawlers interact with websites.
🚀 Whether you’re running audits for clients or navigating SEO for an enterprise site, this conversation is packed with takeaways that will upgrade your strategy.
🔍 In This Episode
• How Sitebulb grew from a desktop tool to an enterprise cloud platform
• The strategic decision that made Sitebulb stand out—hint: it’s all about "hints"
• Why JavaScript SEO and regular audits matter more than ever
• Insight into AI crawlers and how they differ from traditional bots
• Sitebulb’s plans for integrating task management and user-defined segments
• How structured data and content optimization are evolving in the AI age
• Behind the scenes of Sitebulb’s State of JavaScript SEO Report
• The future of SEO in an AI-first search experience
Stay ahead in SEO by understanding the tools and tactics built for tomorrow’s web.
đź“Ž Mentioned Links & Resources:Â
[00:00:00] Jon: How do you build a technical SEO tool that stands out in a market dominated by giants and still say fun enough to sneak a few curse words into your release notes? Patrick Hathaway is the co-founder and CEO of site bulb, a technical SEO crawler that's quickly built a loyal following by solving real problems with surprising clarity.
Our agency Moving Traffic Media started using it in 2018. Just a year after launch, and we've watched it grow from a scrappy desktop tool to an efficient cloud platform for enterprise teams. We dig into how site bulb evolved from an add-on to a standalone platform. Why features like JavaScript rendering and hint based reporting changed user expectations and how cloud infrastructure and AI crawlers are reshaping enterprise.
SEO. We also get into Patrick's take on AI search. The limits of LLM crawlers and why SEO fundamentals matter now more than ever. Patrick's building for an AI future, but doing it with a very human strategy. Stay close to the customer. Question the hype and focus on what actually works. Whether or not you use site bulb, I'll think you'll enjoy this one.
Here's Patrick Hathaway, [00:01:00] co-founder and CEO of site bulb.
Patrick, welcome to the show. Great to have you. And during the research, I was going through the website and noticed my face on the homepage. So needless to say, we are a customer. We love the tool, so I'll just sort of get that out the way. We're probably a little bit biased in this interview, but. I was excited to dig into your past.
You started Site Ball in 2017. We started being a customer in 2018, so I think you could argue we were an early adopter of sorts, but I'd love to hear a little bit about your progression from, so you were running a different tool. You are. URL Profiler, which I'll be honest, I didn't actually use, but it looks a little bit more like a Screaming Frog tool.
Is that a fair comparison?
[00:01:53] Patrick: Yeah. We were building the site audit, well, the site crawl feature to append onto your R profile. That was always the initial idea [00:02:00] and we thought we had a new UI for it. We were kind of designing everything to work around the idea that this crawl would fit into the existing product.
And as we were doing that. We figured out, actually we probably should just include an index ability report. We, we added that in and then we thought, actually maybe we should add in an H ang report. So then we added that in and then all of a sudden this thing kind of grew lag and we're like, actually this is sort of its own product.
And we had done at the time as we were building up your L profile. 'cause one of the things, when you are starting a SaaS business from scratch, bootstraps, as we were, you have a long period of time where you'll only get making. You know, a couple of hundred bucks a month right before you've even got enough money to pay salaries.
And so I was augmenting, uh, the money that we were making through URL profile in the early days by white labeling, doing technical SEO audits. So I spent two years pretty much trying all of the call tools out there. We had subscriptions to all of them. I was often doing things with other people's accounts, or it's basically using them all the time.
And so when we got to the point where we're building our own crawler. Actually, I've seen this in this tool that [00:03:00] was quite good, but not quite right. I saw this in this tool that was quite good, but not quite perfect. What could we do and where could we add value to this experience? You know, it didn't take as long to come up with a number of things, which were very different at the time to completely differentiate site, but from anything else that was out there.
Really. Then,
[00:03:16] Jon: yeah, I mean, running a SaaS business is so different than, you know. Even running an agency, right? Like the decisions and the stream of revenue, right? All, all of those things, there's a lot more upfront costs. And I guess you had a little bit of experience with this running URL profiler, but you were sort of entering a really crowded space, screaming frog, deep crawl.
I mean, I guess you could even throw Z in there, right? Like all these different sort of competing products. And then of course you had the enterprise tools conductor and Link X was around back then and you know, how did you think about sort of entering that market? Were you targeting more of a small or like a medium sized business and sort of leaving?
I. The larger enterprise businesses for some of those other tools? Or sort of, how were you thinking about, you know, competing in that [00:04:00] space?
[00:04:00] Patrick: Yeah, I mean, we saw, so you, you've gotta remember that, you know, this is late 2016, early 2017. Screaming Frog was fundamentally a crawler that would fall over when it got to 30,000 euros.
It couldn't handle it, right? They hadn't yet built the ability to sort of store. On hard disk. And so you had an option as an SEO at the time. You could use Screaming Frog for most of your crawls, but as soon as you wanna scroll anything bigger, you had to go up and pay orders of magnitude more to get a deep crawl or a body buy at the time.
You know, any of these tools which would allow you to crawl the bigger sites and naturally with all these other tools. They had a more user friendly experience. They had things built into them, like reports built for different users. So we tried to sort of plug the gap, right? How do we give something that's a bit more, a bit closer to the experience of something like Lumar, but closer to the price point of Screaming Frog?
And that's where we entered and we were, if we can get some of the people up from Screaming Frog to use something like Site Bobb and some of the people kind of down who might have used Deep Call or Oncall or whatever. [00:05:00] Otherwise that was the sweet spot we were aiming for. And you know, ultimately we've ended up doing the same sort of thing again with Site World Cloud where we are now because we think of similar things.
Ha happened. But that's what the experience felt like at the time. And so when we, you know, we, when we first even launched the beach of Site Bobb, we had JavaScript crawling natively. We had saving directly to a database that meant you could crawl hundreds of thousands of URLs. We had things like the crawl maps, the site visualizations.
These were things that no other tool had at the time. You know, you couldn't do any of these things. Certainly with a desktop product you couldn't do the scale and no one was doing JavaScript crawling, I think Screaming Frog rolled it out sometime in 2017 while we were doing beta. But you know, the big cloud crawlers hadn't figured out how to do JavaScript crawling at any kind of reasonable cost at the time, and this is even before headless Chrome came out.
So Headless Chrome came out in 2018, which kind of changed the game for how the technology was used to do. Java rendering within your crawl.
[00:05:53] Jon: So just sort of like thinking back, you've maybe asked it a little bit differently. Is there a single sort of like strategic [00:06:00] decision sort of looking across that landscape that you think really propelled site bulb?
So for example, was it the focus on JavaScript before sort of everyone or you know, I think you could make an argument that it's moving to the cloud, right? Which now. Is also a pretty unique feature for you guys, again, across the competitive set. You know, if you sort of look back, is there a single thing that really stuck out as the, as a pivotal moment?
[00:06:23] Patrick: I think there's, there, there's one pivotal thing that we did to change how we delivered data to our users. That wasn't necessarily that fueled growth, but it fueled adoption and usage of the tool and has sort of been the thing we hang our hat on. Ever since, like at the beginning, genuinely people would come.
To use site board because they'd seen the fancy crawl maps and this was quite a shareable asset. People could see these, you know, site visualizations that looked interesting and were sort of fun and so people kind of came for those and, but people sort of, I've had this, I dunno if it's quite on the website anymore, but it certainly was for a while.
I came for the crawl [00:07:00] maps. I stayed for the hints, and it's really the hint system. We built the way of prioritizing issues and opportunities within Site BOG that actually differentiates it from all the others really. And it's that workflow that we doubled down on. You know, that we had that from launch, but during 2019, we doubled down on.
What education we surrounded it with, what date, what metadata we surrounded the hints with. Even how we wrote the descriptions so they could be used in like a white label environment. So I would say it's the hints that were the single biggest change that we made that had an impact on how the business act.
[00:07:31] Jon: I love the hints. I think most SEOs probably do. There are a lot of sort of plug and play components to it, right? Or you. Don't have to come up with all that text and those, you know, reasons why it's an issue, how to fix it, all those sorts of things. Was the foundation of those from your days doing audits or was this literally net new content created, you know, on the back in support of site bulb as a tool?
[00:07:54] Patrick: The, the stuff on the website, do you mean? Yes. That, no, that was new. That was six months worth of work for [00:08:00] me. That was literally, you know, here's how I understand the thing. How do I explain the thing? How do we get this across to customers? Through the code with Gareth to make sure we fully under, I fully understood how the hint actually worked and what we were checking.
'cause we have things like code examples in there as well. So it was, there's a lot of what we put into site build was a brain dump initially from me and Gareth from our experience doing SEO for a bunch of years. And what we wanted to include in, in technical audit or technical audits. What's interesting is, you know, the more you work within product, the further away you are from actually doing the work, and the more you have to rely on customer feedback and talking to your customers and you know, finding out what they're doing, what their challenges are, rather than where you start out, which is I can build a tool ostensibly for myself.
There's a phrase that gets used on almost every single SEO tool website, which is built by SEOs for SEOs, right? And you know, we used to have this, we don't have it anymore, and it was true. Right, and it is true for a time period, but then you get to a [00:09:00] point where it's not true anymore because you're no longer really an SEO because you're not doing it.
You are, you become very specialized in a certain section or certain area, but you're not actually doing that work anymore.
[00:09:08] Joe: You spent, how much of your day do you spend talking to SEO professionals? I know, I'm sure it's every day, but is it like hours every day talking to customers?
[00:09:16] Patrick: Yeah, it's most, I mean, you know, this is, ever since we launched Site Cloud and people are.
Essentially wanting to have a sales conversation because the price point is at a point where it's a much more considered decision. We get to a situation where people want to have a conversation about whether or not it's gonna be right for them. So I handle all of that at site. So I do all of the sales calls, I do all of those and might be 1, 2, 3 calls depending on the type of client and how much they need.
Which, you know, is a lot about discovery and a lot about figuring out what their challenges are and where site will be able to plug in. It's part demo where I'm showing them how the tool works and then you know, further along you get with the progress. Sometimes it turns into training and actually. This is how you use the software.
And again, that's so much when you try and customize that as much as possible [00:10:00] for your business, how could this work for you? That's how we try and structure things at Sidebox.
[00:10:04] Joe: I think maybe the nice part about SEO over the years is there's always been this plan that you could stick to. There were a handful of audits that you could deliver.
You could build a program out of, you know, these five or six. Standard deliverables for every client. They were a little bit different for every client, but you almost always started with a tech audit. And the tech audit changed a little bit every year, but the fundamentals of it were the same. I think the big thing that changed for us when we started really using site ball biz, we would do many audits throughout the year.
Wasn't just the, this is our first big deliverable, here's our tech audit. I need two hours of your time explaining it all. We still did that, but throughout the year we would do more technical audits. If they would just be a little bit smaller in scope to focus on things that we tried to correct in month one, that's been a big difference for us with site bulb.
[00:10:51] Patrick: Yeah, I mean, I would agree. I think that's where I, we see agencies doing well is when they move away from that approach that a technical audit is kind of like a [00:11:00] one and done, which as you say, it made sense back in 20 15, 20 16, you know, back in the early days of site Bobb, where fundamentally the landscape wasn't really changing that much.
Certainly in terms of tech, right? You know, you could optimize a site reasonably well from a tech. Perspective and things weren't actually gonna have that much of a, of, of an impact moving forward. 'cause things weren't actually changing that much. But you know, the more regularly you actually order, whether that's quarterly, whether it's monthly, whether it's even twice a year.
It just increased the touch points for going, okay, actually where, what is happening right now? What is the situation on this site? And the more complicated the website, the more, or, well, it's not necessarily complicated the website, but the more different people that impact things on the website, whether that's content creators, user generated content, multiple different tech teams, the more.
Greater the chances that something will go wrong somewhere along the line. And if no one's checking for those things, then you know, you don't even know what you don't know.
[00:11:53] Jon: I think the amazing thing is a lot of people don't connect that once you implement a tech change, [00:12:00] there are orders of effect that happen across the side once those things are implemented.
So even if you just run one big audit and implement a bunch of things, it doesn't mean that the site is clean. It could have created something new. So if you. For example, expose an area of the site to Crawlability that wasn't available before. There could be a whole new subset of opportunities, you know, within that section.
And so I think the ability to set up the scheduled crawls where we're looking at things weekly has really been effective for us as, as well. I wanted to circle back to the notes real quick, more of a maybe funny question, but was it hard not to curse when you were writing the notes for the issues, the release notes?
[00:12:39] Patrick: I mean, no, I naturally, I enjoy. All language. I try my best to insert them wherever possible. There. There was a time right in the very beginning, like I think probably even during beta when I would email these out and I got one response one time where the, where a guy said to me that this is not professional to be swearing in your release notes.
I don't appreciate this [00:13:00] so unsubscribing and that knocked my confidence. So I stopped in the next one and then I got, you know, 20 emails from people going about, whoa, hang on, what's going on here? What, these were fun. Why have you stopped being fun? So I was like, okay, alright. 20 to one. I can take the hint here.
That was enough encouragement for me to continue putting as many curse words in as I wanted.
[00:13:20] Jon: I mean, as funny as, I don't know, non-traditional as it is, it's sort of like the. Voice of the brand in a way. That's what people sort of, you know, appreciate about the release notes. You're not only releasing great things, but there's sort of a.
A funny tongue in cheek way of presenting it. We talked with Zach JOAs last week and we talked a little bit about structuring technical like notes or dev tickets. And one of the things I liked about the section that you built out with all the helpful insights for all the issues that are identified or the way they're structured, which is very similar to, you know, one of those tickets, I was curious if you ever thought about building.
You know, a [00:14:00] ticketing component into site bulb where, you know, it would just automatically grab the URLs, create the ticket maybe based on that library of content, and then you could submit that right out. That's probably a, maybe an edge case depending on the size of, you know, the company using the tool.
But I. It seems like you have a lot of the components there and a lot of the structure already. It would be a nice way to sort of immediately inject that into the workflow on the dev side.
[00:14:23] Patrick: Oh, absolutely. I mean, it's been on, it's been on a, a roadmap, been on a backlog for a while. I recently got it onto the roadmap because I've been campaigning for this internally for a while.
I feel like certainly for some agency teams, but a lot of in-house teams, it's the natural next step. I think though, to make it really work, you need to really align with how users actually want to get this data out. 'cause if you just build a very complicated task management system within the tool itself, that is almost like a mini Trello or mini Asana within the tool.
I actually don't think you'd get the usage adoption. I think it really needs to plug into something they're already using. For it to really [00:15:00] work. We are, we have, we already made progress with this, so it's something which I hope we'll be able to release this year. There's quite a lot. I think there's quite a lot to it, even though we've got quite a lot of the bits and pieces that could generate it.
I think there's quite a lot to that. That end piece that I was just talking about, like making it. Work as seamlessly as possible with whatever platform that the user is using.
[00:15:21] Joe: Can you share any other ideas that you hope to launch this year or We're halfway there.
[00:15:26] Patrick: Yeah. Yeah. We've got, so the big features we're working on at the moment that one, one of them I think is a bit of a gap in our product, which is user defined segments.
You know, we have a HM HTML template feature, which automatically tries to pick out and group pages based on the underlying HTML template. But site is currently missing the ability to. Group pages as the user. So if you want to group your pages to do with TVs and to do with refrigerators, whatever, based on page path or some other component that is on the page, you know, pretty much all the competitors have got that.
And so I think it's something that we are missing. We are, we have [00:16:00] become more interested in log file analysis for two reasons really. You know, with Site Cloud, we are now dealing with enterprise sites. We are dealing with in-house brands that. That rely on and care about this data. So it's more important for them, which I, I always think about log as more, more important for in-house than for agencies, really, typically.
However, there there's another potential need for log file analysis and that's what the heck are LLM callers doing, right? And, you know, how do we answer that question? So I think increasingly, you know, if we do genuinely see chat two BT and the others, you know. Encroaching into, into Google's, uh, market share.
It will become more and more important. It's very, a AI is very much a buzzword at the moment, you know, it's. It's the new interesting thing that's changing in the industry. I don't think it's necessarily going to have the impact that some people maybe think, but I can see why everyone's talking about it.
You know, fundamentally, if it's data our customers need and want to get out, then it's gonna be something that, that we're gonna look to, to do. And [00:17:00] then, you know, finally there's things that we can be doing in terms of vector embeddings that will help users better understand. How LLMs are seeing the data.
So it's a little bit of a shift, I suppose, in what sort of data you might want to get out of the tool, like site build, but kind of shows you the direction that we're thinking.
[00:17:17] Jon: So to dig into that a little bit, are you thinking about building a crawler that would emulate like a Chatt BT crawler, or are you more thinking about components of a page?
That would need to be discoverable if, you know, an LOM crawler came into the site, for example, JavaScript, right? Like we,
[00:17:35] Patrick: yeah, much more the latter. You know, I, I, I've, you might have seen me on LinkedIn sort of bashing this idea that an AI crawler is even necessary. Like, why do we need an AI crawler to go and look at a page and tell you what the page title is and what we figured this out literally decades ago.
You know, even Zeno could do that. We don't need to, to reinvent the wheel, but where. Mentioned JavaScript. I think that's a huge part of it. [00:18:00] Understanding that, that actually the large majority of the LLM crawlers aren't rendering JavaScript, so having content that is effectively hidden to them because it's only accessible or changed during JavaScript rendering is a huge problem, which is, you know, we have insight, but we have a response versus render report.
Which allows you to kind of see these things. So there's, we've kind of taken care of that to a large extent, as long as users connect the dots between that content. But what I'm really talking about is helping users to understand what they might need to do to their content to optimize it for how LLMs understand different chunks of content, right?
Because they're not necessarily considering pages in the same way that, that Google traditionally. Did. It's much more about that sort of did little chunks of content and what they mean and so optimizing the passages so that you've got, you know, the ability to understand what's going on and what you need to do to optimize further for, uh, l LMS to synthesize the content.
[00:18:57] Jon: Yeah, I mean that's super interesting because I think a [00:19:00] lot of the content output. Of the tool today is mostly around traditional like metadata. There's some duplicate content review and ability to extract, you know, certain components of a page, but really digging into the structure of the content itself I think would be a, a pretty cool evolution of what you've already built.
[00:19:18] Patrick: Yeah, I see it coming kind of full circle in some ways in that we, you know, we just mentioned that, that JavaScript, you know, removing the dependence of JavaScript for your core components is important, so. It's a little bit like, are we back into sort of 20, 20 15, 20 16 when all the advice was don't have any JavaScript, and what does it look like in the source?
But also, and, and I know you guys do these at Moving Traffic, is that the, the content audit is, is suddenly valuable. Again, like I think this was a really valuable thing and it's actually one of the reasons we talked about UR l profile before. One of the things that people would do with your L profiler is pull together these content metrics, do these content audits using a combination of crawl data and the data you could [00:20:00] get out profiler, and I can see content audits being another deliverable that comes back with the perspective of.
How easy is it for LLMs to understand this content topic modeling, you know, are the topics on across the site consistent? Are we linking between topically relevant pages? So all of that stuff, I think, you know, becomes, yeah, kind of comes full circle and becomes important again.
[00:20:24] Jon: It's kind of amazing. I was reading something the other day that was talking about, one of the chat GBT engineers was talking about how they discover content or how they rank content when they actually crawl the web and it was.
Basics. It was like the freshness of the content and even down to the relationship between the meta description and the content itself. And we've talked about meta descriptions as not being a ranking factor forever, and now you're sort of coming back, like you said, like back to simple basics of what SEO was years and years ago.
So it's really as [00:21:00] revolutionary as AI is in many ways. It's also dumbing down. Where SEO has evolved to, right? Because we almost have to do things that have become less important. You know, simpler technologies like straight HTML, meta descriptions, things that we haven't really spent a ton of time on.
Yeah, no, exactly. On that basic level
[00:21:18] Patrick: and you know, the structure of the content. Which, you know, we've been taught for years to think that the H one's important, then everything else. But you know, anyone who like values the readability of a document will go, I wanna have consistent headings and I want everything to fit under HH threes.
And so if you've been doing these things, then that will automatically make it easier for LMS to understand what's going on. But then like you say, you know, making sure that you are refreshing your content to us up to date so that it's got even citation links, right? So it's got like all of the things which Google's been trying to teach us in terms of EAT, right?
So all of those bits with authors and anything that you can add in [00:22:00] structured data to help. Reinforce that pages are sort of ticking all those boxes that, that the LMS are looking for.
[00:22:07] Jon: Yeah. I did wanna just jump into structured data. So you guys do a lot of great out of what you're discovering during the crawl around structured data, and I think all the search engines and AI models have come out and specifically said that, Hey, we do use structured data despite all the maybe debate that there's been online.
Are you guys thinking about doing anything? Additional there to sort of elevate those insights or I don't know, even suggest maybe opportunities there or,
[00:22:33] Patrick: yeah, so that's, I think the structured data report inside site, Bob is fundamentally based on discovering what's there telling you, where there's gaps, where there's thing's missing.
In terms of the breadth across the whole site, but it's, it's fundamentally a validation tool, right? It will tell you, and I think Site Bob's structured data report is, I would say, the best on the market in terms of helping you understand where the issues are because it aggregates them to date together, so they're all nested together.
So you can just click through it and you can see. [00:23:00] What it doesn't really do at the moment is, like you say, kind of give you those opportunities for this page, or these pages are missing. X, Y, Z, which could have an impact. So that's, that would be the next logical step we would take take with that feature.
What we have focused on really with a structured date report is. Making it really easy to use and really easy to link all it, kind of nest all the data underneath it one another, but also staying on top of all the changes. And I dunno if you've seen the recent Google announcement that they are sort of deprecating a horse wave of rich results.
So that is certainly one thing that, that I think we, we will see is the more that Google focus on AI mode, the less important it's going to be that. They've got rich resort X, Y, or Z, because they're replacing that experience with something different with the NA mode.
[00:23:45] Jon: Interesting. So markup within the search result itself is less important because the structure of the page is fundamentally changing the, the ERPs.
Yeah. Yeah. It is changing. Yeah. Yeah. Makes sense. You guys released, speaking of JavaScript, you guys released a, like a [00:24:00] state of JavaScript report in 2020. Are you guys planning that for 2025 as well?
[00:24:03] Patrick: We are, yeah. So we did a whole study which used a combination of survey. Or users, customers to, to get their feedback.
And we also looked at data collection from site world about usage, the types of errors that we saw, all that sort of stuff through 2024. And we planned to do the same in 2025. The thing we did actually off the back of the report in 2024 was basically we, we realized, or one of the big takeaways was that there was a huge gap in knowledge and understanding about.
People think it's important and people feeling confident that they actually know what they're doing. And so then we work with women in tech, SEO to put together through, you know, women tech, SEO partners. We put together with them a course ran by Sam and Tori from Grade Company, which was a three part course on training Forger Street, SEO.
So this is available on our website that you can go and download. We. Provide the links that you can look in the notes, but basically it's a really good starter for anybody who's uncomfortable, [00:25:00] unfamiliar with a, doing Atory, SEO, but also actually communicating it. And I think that's the big PO point, right?
Like you can, a tool can tell you, oh, that, you know, this thing's changed or that thing's changed, so. Seeing it is one thing, understanding it's another thing, and then actually being able to communicate is another thing again, and like the most effective SEOs are the ones who can communicate what needs to be done and why in a way that actually encourages people to take action, which is a bit of a mouthful, but there's a lot going on there, and I think this course is a really good primer for that.
And so what we really hope to do with grad is work with them again, to kind of do the next evolution of that course as well. As we see at the study for 2025, we'll also include questions about the impact of how Java fit works alongside ai. So there's Evolu, there will be evolutions in what we're asking, what data we're trying to get in the first place, and then hopefully we can also follow up with a, with sort of part two.
[00:25:54] Joe: It seems like the conversations around optimizing for the JavaScript on a page are some of the hardest [00:26:00] conversations that. SEOs have to get into, you need a, like a developer partner who's willing to share with you, what was the intention of the JavaScript to begin with? What does it work exactly? Yeah.
What are some alternatives to it? You can't just produce an output and send it over. That has to be a tough conversation. You need a good partner for that.
[00:26:18] Patrick: Yeah, no, for sure. And like it, it comes back to being armed with the enough understanding and information about what the issues are and what's causing them to feel confident to open up that conversation.
Also to feel confident saying, look, I know something's going on here. I don't necessarily understand what it's doing, but this is the SEO impact it's having. And you may not realize that. And again, having the confidence to be able to outright say, I don't know what's going on with this thing, but it's having this impact.
Can we discuss how we might be able to do this differently?
[00:26:49] Jon: Yeah. I came across a post from Mark Williams Cook, who always puts out great, you know, unsolicited, SEO tips, and it was basically around the. Unused CSS and unused [00:27:00] JavaScript that's reported in site bulb. And the tip was basically like, Hey, there's no tool out there that you know, interacts with the page.
And so if you're reporting against JavaScript that's unused, it may actually be used when someone, you know, expands an accordion or clicks a tab. Are there any other tips like that within the tool that you maybe are less well known? It doesn't have to be specifically with JavaScript, but maybe just. Tips within insight bulb in general.
I think every time I reach out to you with a question, you always have some great insight that I didn't know about the tool. It's just so there's so much in there. Right. Are there any common themes that jump out at you as far as you know, things users might not know? I.
[00:27:40] Patrick: I mean, that code coverage report is the thing you're talking about is, is one of those tools that we have some power users that really get a significant amount of value from it, build their businesses off this thing because it's really powerful.
It, it kind of requires a relationship with where there's a lot of trust and like a deep relationship between the SEO [00:28:00] and the tech team and like a willingness to wanna improve performance, which I think go also goes with, you know, call web battles, right? Like you have to have that. Have to be on board that they, there's actually something that they wanna do there in terms of features, insight, bobb, you know, outside of the code coverage.
One that's, that sticks out for me is the single page analysis. And so this is literally the ability to just take any single URL, you plug it in. And this is available both on our desktop version and our cloud version. Site board essentially run every check in site board on this ERL, including the, you just talked about, including collecting web core, web vitals, all the basic SEO data, internal links, images, everything.
And also including the JavaScript piece. So what's actually on this page, which is render by JavaScript. What's JavaScript doing to the page? And so when you are doing, you know, if you're looking at implementation or even if you've just got a quick spot check, you just wanna check something that a client said.
You can just put that and just check it straight away. So it's a really quick way to. Validate. It's a really quick way to, again, just spot check things and just [00:29:00] see, actually is what they're saying accurate or can I just get a quick read on this page before I even dig into, you don't always have to look through hassle or Deb.
[00:29:08] Jon: Yeah, you actually turned me onto that, that, that feature, I dunno, probably six or eight months ago and we started using it in our, our schema audits. So we would, we do. Crawl, you know, the client side. And then we would take a similar page template from a competitor, drop that in there, get the output and sort of compare, you know, schemas across a cohort of competitors and sort of see where those gaps are.
And that's one way that we sort of helped identify where there are some of those opportunities. So I guess, you know, if we look forward a little bit, you know, the industry is changing quite a bit. Are there any, you know, new maybe technical. Challenges that you could anticipate? I don't know, coming out of AI mode or the way that LLM crawlers are navigating websites, anything new that you could anticipate that you'll need to sort of work into the tool?
[00:29:53] Patrick: Probably not beyond what I've already shared. I think the thing that we only need to remember AI is that we are [00:30:00] right at the very beginning of this thing. Like this is gonna fundamentally change how everybody searches. We don't even have AI mode in the UK yet, right? Like we, that, you know, you guys must be used to it by now, but it's not even rolled out across the UK yet.
And it is going to completely change the way in which we have to think about SEO as a whole. And so I think going back to those basics is actually, you know, still gonna be really important. Everything we, we mentioned before about the value of a content order, um, the value of getting your technical SEO basics, right?
And you know, one thing which we have learned from some of the studies that have come out. The, the sort of frequency of appearing in AI results is very heavily correlated with rankings, right? So if your rankings are pretty strong and steady in, in traditional search, it stands to reason that they're gonna move forward in, in AI mode as well.
Um, but also like understanding that, you know, pages in the same way that they aren't gonna end up in the Google index. And that's, you know, a, a barrier [00:31:00] to, to being in the search results. We have the same barriers for. To elements, right? If they can't crawl the pages properly, if they can't. Look at, see the data properly.
If they can't actually index the page, then it's not gonna end up in the results. So that, I think a lot of the, I think we're gonna come back to a lot of the fundamentals. And then in terms of what actually, genuinely I is gonna be new. I think as much will will come round to SEO's. Understanding the pages that might have worked well for search previously might not necessarily work well for LLMs.
So have they got a clear summary that makes it easy for their AI to understand what's going with the page? I, I mentioned, you know, the sort of trunk retrievable stuff before. Is there different ways in which LLMs are interpreting the data and essentially going back through and Retiming pages that may have worked okay from a search perspective to also make them work for.
The likes of chat gpt,
[00:31:54] Joe: you're talking what you're It's the content though. That's the big, that's the big like question right now is [00:32:00] it's, there's some content that AI is probably a better experience for informational content. You're trying to understand something very high level or if something very deep, but the transactional things still require people to spend time on a website and they're not, hopefully the behavior, you know, moves along where people don't rely maybe.
With agents in the future they will, but for the most part, the transaction still needs to happen with on the website.
[00:32:25] Patrick: Yeah, for sure. And I think that's what we are gonna see is that, you know, the, while the transaction still happens on the website, all of the research kind of happens somewhere else, and so we don't get to see these sessions that we previously saw where people may be coming on and viewing some of our top of funnel content.
Actually, that content is getting ingested and it's getting served elsewhere. Attribution is gonna get even more difficult, right? So how do we know what content we should be, what we should be producing? I saw Matt Barbie did a post on LinkedIn yesterday, I think, which was really interesting sort of thought model around this, which was, you know, the [00:33:00] idea that you should have surround sound, right?
So you mentioned in as many results on the first couple of pages, used to use it at HubSpot, he said in his post, I think, but that you, he would try and do this with the traditional search, but also the same thing. For LLMs, right? So if you are getting cited link or no link in lots of the results for your, the sorts of queries and conversations that the people are having, that's gonna have a huge impact on that top of funnel research.
You know, hearing your name again, whether or not they click through, right? Just learning about you. So that when they are later on in a place where they do wanna buy or they do wanna have some sort of transactional purpose, that brand awareness is there and they do come through to. Hopefully she come onto to your website.
[00:33:46] Joe: I, I think we start to develop new strategies for growth. One is around informational queries and the other is around purchase intent queries. That's, I mean, maybe we've been doing that all along, but I think it becomes even more important now when you [00:34:00] think of where infor, where the, the searchers journey starts.
Maybe it's Amazon, maybe it's Chat GBT. It's probably still Google for eight outta 10 queries, but we can think about those different. If those different, not so much the path that the customer goes. From discovery to purchase, but what platforms they rely on for the different types of queries. It's a, it is kind of a different way of thinking.
You're not only optimizing for Google anymore, so you kind of think about where is the customer at this point in their journey? How do I optimize for that point? We're not gonna be called SEO people I think in three years, and I like, there's gonna be a new name for us and I don't know what it is. I'm sure you've heard a hundred of them, but we're not gonna.
Just be SEO.
[00:34:43] Patrick: I have had hundreds of them. I dunno if I agree that we won't be SEOs. I think what, what constitutes a search engine perhaps will just change so we can keep our name. But I mean, I totally agree with you. You know, understanding the bio journey is now one of the most important, important facets of the role and [00:35:00] producing content.
For each of these phases and making sure you've got that. What I think is gonna be really difficult is how do we justify doing that work when it's not, there's no attribution for it, or it's very difficult to get attribution for it because it's still important. You create that content so that you get cited and people learn about you.
But actually, you know, figuring out if it has any impact further down the line is gonna be much, much more difficult. I think, you know, whether we can get budgets for that sort of work, who knows? I think we're gonna have to have tools that help back up that this sort of content does generate citations or just does generate.
Mentions in LLMs that you can at least indicate there's a brand value to this.
[00:35:38] Joe: Is there something in the log file analysis that big companies can do because they have access to such great like
[00:35:45] Patrick: BI tools? Well, you can tell that, you know, log data will tell you whether the. LLM call have accessed your content, but not whether it's actually being used or appreciated or appearing anywhere.
You know, I'm sure crawl frequency would give you an indication that [00:36:00] of page importance. But even then, you know, I dunno if you saw this study by Sal along with Merge whi, which was I think the first evidence that we saw about LLMs not rely not being able to execute JavaScript. A a lot of the things that, that they were talking about in their study was that.
These AI crawlers are so inefficient. You know, they haven't developed these really efficient algorithms that like Google have to go. Actually, we know that page is a 4 0 4. We're not gonna visit that again for another month, but we'll keep it in the index. We'll hang around. We wanna double check that.
It's a 4 0 4. They're just randomly hitting stuff. They're hitting things. Making up strings sometimes, you know, they're not using crawl, their own crawl budget very efficiently. There's also been studies out there that have shown that the chat, GBT, uh, OpenAI bot is hitting sites, you know, 12 times what Google Bot is.
Right? And if that stuff continues, at some point, certainly bigger companies are gonna go, do we actually, are we happy about this? Can we rate them at these? Can we throttle these in some way? You know, it feels at the moment [00:37:00] that everybody is only seeing the upside of ai. Actually, I think there's also a downside, a from bandwidth, right?
It's also like massively not eco-friendly to be, to be just run, you know, crawling random four oh fours that you've just made up. And like you say, the cost implication of all of that and the servers you uptime that you need to make sure that your site remains performant. But also the idea that. We want crawlers.
We want LLMs to access absolutely all of our content, and I just don't think this is true. I think that we will come up with ways and means in the same way that we do for search engine where we go. Disallow these pages, no index those pages, canalize these pages. The, we essentially say that we don't want the LLM to crawl this stuff.
It's not, it doesn't add value. We, or it might even be factually wrong, right? It might be old historical data that we know is important for us to keep on the website because we need it maybe for a legal requirement, right? But we don't actually want the LMS ingesting this because they might use that in a response that's not up to date and is not accurate.
So I think [00:38:00] that will change, I think how we. The view, what we want to allow LLMs to access will shift over time, and maybe we'll end up with some new protocols like LLMs dot xt, if enough support comes behind it or there'll be directives, you know, like robots based directives that we can give them that they might actually follow.
[00:38:19] Jon: That's what I was gonna say. Like they, right now, they bypass pretty much any robot stop txt command. So the ability to even. Throttle them or limit what they're getting into. Sometimes they just blatantly ignore. Anyway. We've had a couple clients where the default is just to block everything because they're hitting the server so hard and then they want to know why they're not showing up at all in LLMs, and it's like we gotta choose one or the other for now.
The risk is getting, it's difficult
[00:38:43] Patrick: balancing actually.
[00:38:44] Jon: Yeah, for sure. Yeah. And like
[00:38:46] Patrick: you end up like a lot, like they probably did in that instance, if they don't respect robots T xt, you're like, okay, let's go onto CloudFlare, right? Let's go on to the firewall rules and we'll block them there. And you know, I dunno if actually it already exists in CloudFlare.
Literally [00:39:00] just a button to go, I don't want these guys, and just block it that that way. I think maybe it does, but I can certainly see that the big CDM providers putting these things in place. Because it will start to become a problem and you can't just go around, you know, doing whatever you want. There will come a point where this is not okay.
And you know, there is, there are actually some ramifications, which it doesn't feel like there are at the moment.
[00:39:20] Jon: You mentioned something around links and I know that I. There are plenty of other tools that do, you know, link data quite well. Is that ever
[00:39:27] Patrick: anything that you guys considered internal of in internal inbound links Like HRES and Majestic?
Yeah, yeah, exactly. That, that, you know, you are talking about crawling the web, right. To do that sort of thing. And if you've seen any of the, the hres have put up over the years or m have put up over the years that, you know, show the amount of investment they've put into their data centers now I don't think we need anymore.
Inbound Link callers. We've got good ones already. That won't be I, you know, fundamentally what it comes down to for us is we don't wanna become a suite, right? We want Site Bobb to be the best [00:40:00] technical SEO caller, we think Point solutions you essentially better data. A better price point. You know, a lot of the suites, they sort of cover a lot of bases, but don't do any one thing especially well.
So that's not a direction that we really wanna go with a product outside of the sort of cost implications of building the thing in the first place. But yeah, you know, for us we kind of wanna stay in our lane and do that one thing really well. Makes sense.
[00:40:22] Jon: That's probably a great place to, to wrap up, but we do like to ask one sort of prediction question.
I know we've asked you a little bit about the future already, but if you could imagine going to google.com. Entering a query, what do you expect that experience to look like in, you know, a year from now? It
[00:40:39] Patrick: depends. Does that count? Well done. I mean, like I said, you know, we are still in the uk. We don't have AI mode yet, so I haven't really had a chance to properly play with it.
I've seen what, what it looks like through videos and stuff that people have shared. I think we are going to see Google over the next year iterate that experience as much as they can. You know how much they love testing. [00:41:00] I expect it's gonna be present for a whole chunk of queries and I think they are gonna have to figure out and make sure that they have, uh, an answer for their monetization model.
So what that looks like, I think is probably the, what Google looks like is probably most based on how do Google make money and how do they integrate paid with AI mode as to what it will actually look like. I dunno, I don't have any more insight than anybody else, but I think, you know, we saw the results of the DOJ trial, right?
How much of their revenue. Purely come from PPC. So it is going to be a huge challenge that they are definitely working on to make sure that they can monetize ai. So some sort of bastardized AI version, AI mode with, with the PPC in there somehow is probably my best guess to, to what it'll look like.
[00:41:49] Jon: Amazing. All right, Patrick Hathaway, everyone. And remember, if you like the show, please rate, subscribe, and comment. We'll see you next time. [00:42:00] Bye.