Welcome to Press This, the WordPress community podcast from WMR. Each episode features guests from around the community and discussions of the largest issues facing WordPress developers. The following is a transcription of the original recording.
Powered by RedCircle
Using search on WordPress can sometimes be an awful experience. On my own blog, I often have better results if I sign into WordPress and search posts through the dashboard. But when I’m visiting other WordPress sites, I don’t have that ability. Hopefully, I can’t sign into their dashboard and search their posts.
So sometimes, I’ve found myself going back to Google and typing a website’s URL in Google’s search bar, along with the specific thing I wanted to find on that site. That’s not ideal.
Today, I’m talking to Luke Patterson, a product manager at WP Engine, and Jared Hutchinson, a senior product marketing manager, about the problems with the default search tool on WordPress and how we can fix them.
I want to say welcome to both of y’all. Y’all are working on a tool that’s going to improve search on WordPress. I’d like to start off though, by asking how each of you got into WordPress. Jared, let’s start with you.
Jared Hutchinson: My WordPress origin story is probably not a fancy one. I started working at WP Engine, mostly, and that’s where I really got into WordPress. Previously, I just had cursory experience, but that’s been my experience so far.
DP: Right on. How about you Luke?
Luke Patterson: So for me, I came to WP Engine from a company called cPanel. So cPanel was my introduction to WordPress because they had kind of like that one-click WordPress install and a ton of people used it. And so when I went from cPanel to WP Engine, was really my opportunity to get really deep into the great world of WordPress.
DP: Yeah. Now that we know how y’all got into WordPress and your involvement there, I want to talk about what I kicked the show off with, which is search and the problems with search. Let’s start off with the first thing I mentioned, which is just my experience with search when I think about WordPress, is that I oftentimes have a better search result in my dashboard than I do on the front of the site. And I’m wondering if, one of y’all could maybe explain to me why that is. Luke, why don’t, why don’t you take a stab at this?
LP: So this is a really interesting one. What may be happening, is your admin search may be searching things that are not available on your front end. So it may be searching for, like, some private data or something like that, or something that your front end just isn’t configured to display. There could be lots of reasons, unfortunately, so that inconsistency in search results, I think, is a big reason why folks tend to turn to third-party search solutions when they’re doing WordPress. A thing that we say all the time when folks are like, hey, why doesn’t this search for this as we have to do the old product manager fallback, which is, oh man, it depends, because data can be weird on WordPress. Right? And when you search, you’re searching your data set, and, data can be weird because of the nice thing about WordPress, which is it can be whatever you want it to be.
So anybody’s implementation or data set or data structure, it can be completely bespoke, pretty effortlessly, actually. And so, turns out, bespoke data tends to break that default WordPress search experience.
DP: And I guess I should mention here, because we’re, we’re talking about this bespoke experience. There are a myriad of plugins to help fix or allow you to customize your search. When I’m talking about search, for the most part, I’m talking about, just the default search. It’s not even a plugin. It’s just a default search tool that is built with WordPress that I’m using. And so that’s on me. I should maybe be considering using some of these other tools. And then when I’m talking about search results, I think one of the things that comes to mind is that. lack of chronological, I want to say lack of chronologicality, but I don’t think that’s the word, but like, you know, like when I do that post search and in the dashboard, I feel like I see here’s, you know, going most recent going backward. And when I do the default search on WordPress, it’s just—I don’t know what the logic is that it’s doing. It’s such random stuff. And it’s only, of course, four results, maybe there versus on my dashboard. It’ll show me. Every single post I’ve ever written. So I guess that’s, that’s an obvious kind of thing.
Jared, why don’t you, why don’t you tell us a gripe you might have with search, like a thing that you’ve come into with search that you’d like to fix or that you think you’ve fixed?
JH: Yeah, absolutely. So, for context, I was actually on our sales team for a number of years before I moved into product marketing. There are search issues across all different types of sites, I think, the number one that I see that Smart Search specifically solves is just with media sites in particular, is the lack of waiting. So, when you have, you know, any number of content on there, you know, all different types of posts, you name it. I’ve worked with blogs that have been around for 10 years at this point. And the number of posts that they have is quite high. And so, there’s all different types of content. As someone behind, you know, managing this site, you want to control, okay, what is my best content? What is the stuff I want to surface? You can control and weight that with Smart Search. I think default, WordPress Search gives you absolutely zero control. And that’s sort of what you were alluding to, Doc Pop. And, you know, that’s what we’ve heard so much from customers is, you know, my search is random. I have no control over the search experience.
And the way that we think about it is, If someone is searching on your site, they are your highest intent user, they are your best user in a lot of ways, and they’re having the worst experience because it’s just totally random. And so that’s, that’s at least the one feature, the lack of feature that native WordPress search does not have that was the biggest issue with our customers.
DP: Yeah. You also mentioned another thing. My blog has been around for 15 years, and so I have thousands of posts and also media attached to every post. So it’s, you know, a little slow going through, but I think it might be the quantity that I have. And Jared, since you mentioned Smart Search, I don’t think we’ve actually specified that yet. Why don’t you tell the listeners what is Smart Search?
JH: So Smart Search is what WP Engine has built to solve this search issue with traditional WordPress sites, monolithic WordPress sites. So as we mentioned, there are a lot of issues with native WordPress search, right? It’s not typo-tolerant, it is not scalable, and it doesn’t allow you any control over the search experience for your users.
What Smart Search is able to do, for all traditional WordPress sites is give you more control over that experience and do so in a really fast scalable way. So you’re giving, you know, the best users, the most high-intent users on your website, you’re giving them a really modern search experience that either helps them convert, helps them stay engaged, helps them find the content that they want to find on your site.
DP: And just to clarify for our listeners now, Smart Search is a plugin, a WP Engine plugin, and it is not fully rolled out yet. It is still going out to just some users at the moment.
JH: So it’s actually, it’s, it is available for all of our premium customers. Uh, it is something that, you know, you can purchase as an add-on if you’re on one of our premium plans. Yeah, it’s, it’s fully available.
DP: Okay, cool. So, In the course of working on this plugin, y’all have learned some cool stuff, and I’m sure we’ll get to that later. Before the show started, Luke, you had mentioned the issues I have with search are not necessarily the same issues that you have with search. In particular, you talked about customer intent. Do you want to give us a quick, explanation of what you mean by that? And then I think we’ll dive into that deeper after the break.
LP: Yeah, so, I am lucky enough to be the product manager for search here at WP Engine and so what I do all day, everybody is talk to people about search and think about search and chat with our brilliant engineers who work on search all day. And so the problem that we circle when we talk about search is the problem of intent.
Why is this person on my webpage? What do they want? And so on the kind of far end of the intent scale, if we are able to completely close the loop between our website visitor’s brain and the computer, so that there is just an instantaneous fulfillment of conscious or unconscious intent, you have like this Neuralink situation where your brain is just plugged into the computer and the website just gives you what you want, effortlessly, immediately, all that stuff. That’s of course like always where we’re trying to get, is to understand, why are you here? What do you want? Even if it’s like conscious or unconscious intent, right?
And so search kind of sits in the middle of that scale or a search bar does. And so when you think about a search bar, what you’re really getting here is someone coming to your website, and explicitly telegraphing their intent. They’re typing in, here’s the thing that I want, into the search bar, hitting return, and then, as a website owner, it’s your job to interpret, or have something interpret for you, that text, that intent-encoded text that went into your search bar, and serve up the thing that, you know, you think best addresses that intent.
And so, we talk about understanding and responding to intent as kind of the central problem of search. And so, you have an ENCODE. I’m gonna type the words into the search bar. And a DECODE. I’m going to interpret that search phrase. And that’s, that’s kind of the reason that folks have different search experiences and also the reason that I want to go back to your point about kind of weighing and, oh, there’s, you know, the most recent results aren’t appearing first.
Like I said, I talk to folks about their search implementations all day, every day, and there’s just a myriad of different things that people want from websites. And so there’s a myriad of different ways that folks are going to want to implement search and how they displace and filter search results so that, hey, I’m responding to my unique kind of visitors’ intent in a way that they’re expecting in a way that serves my business in a way that, right, is up to standards and practices, and accessibility and all that stuff. And so folks aren’t just going to need features like weighing by post or title or date, they’re going to need some interpretation of recency, or maybe recency by popularity. They’re going to need things like, I want the ability to filter these t-shirt results by small, medium, large, or to aggregate some results, obviously to parse natural language.
Searching for Nike trainers is very different than somebody coming in and searching for, hey, I tend to over pronate when I run. What kind of shoes should I buy? And so, folks need to understand their visitors, they need to understand, you know, what is this website trying to do, and they’re going to need a lot of really smart tools to craft an experience that both can understand and respond to with some nuance the intent of your most valuable customers who are on your website and saying, hey, I’m here because I want something, let me type that into your search bar.
DP: On that note, we are going to take a short break, and when we come back, we’re going to continue talking about search and WordPress, and I’m going to drill down into this idea of intent with search. So stay tuned for more. We’ll be right back after this short break.
Welcome back to Press This, a WordPress Community Podcast. I’m your host, Doc Pop. Today, we’re talking about search with Luke Patterson and Jared Hutchinson from WP Engine, who are working on a tool called Smart Search.
And right before the break, Luke was talking about customer intent and, you know, everything I was saying before about my issues, I think it does kind of come back maybe to customer intent. I was talking about maybe slow search results or, you know, lack of chronology. And Luke, as I kind of understand it in a, in a purist sense of like customer intent, someone might come to a site and they might be searching for something for sale in the WooCommerce section of the site. Or they might be searching for a blog post. You mentioned shoes. Maybe there was a really cool post about, how to tie shoes. And so they type, you know, tying shoes, and instead, they get, shoelaces in the WooCommerce site or vice versa. Maybe they’re looking for the item in the store and instead, they might end up finding just a bunch of blog posts that might not be relevant to them.
Is that maybe a way to kind of talk about one of the challenges of customer intent?
LP: Oh, and it actually is a really great challenge to talk about because we kind of briefly mentioned the beautiful ways in which WordPress is idiosyncratic, right? And that means like you can make your WordPress what you want and it’s open source and there’s all these plugins. What that also means is, there’s a lot of ways to modify and tag your data. So with that WooCommerce example specifically, a really popular way for folks to solve this is with something like Advanced Custom Fields, right? I’m going to have some custom taxonomies. I’m going to have some custom fields. I’m going to have different custom post types.
I’m going to have different ways to differentiate what is essentially data between, for instance, blog posts and products, right? And I may create some relationships between the product, i. e. shoes and shoelaces. And so, if you’re searching on WordPress, your search engine is going to want to speak the language of WordPress. Custom taxonomies, relationships, post types, custom post types, all the interesting stuff about WooCommerce. It’s going to need to speak that language. Nobody really does that out of the box, right? Because there’s kind of an infinitude of ways that folks can customize. And so, creating a search experience that, without too much effort on the end user, like without too much custom code, can handle, hey, I understand how WordPress generally treats relationships, or, this is what we do, we’ll say something like, hey, let’s pick a really popular plugin, ACF, for example, WooCommerce, for example, right? And we’re just going to effortlessly integrate with those things, and it’s gonna kinda just work.
And so, we can do other things like, hey, we’re kind of respecting any natural filtering. that comes through from the WordPress side and so building search for WordPress is going to involve a deep understanding of WordPress and the ways that folks WordPress and kind of the naturally bespoke ways that data is going to exist on WordPress so that it can know, hey, if I’m searching for shoes, aha, that’s a product. Okay. And there are things related to this product that I may want to display as suggestions. And so, I could talk more about, I could talk for a long time about search, but I could talk more about, okay, now you’re getting into the way that these results are displayed, hey, do we want to display maybe a somebody search for shoes? Ooh, okay. I want some custom search results such that, alright, I’m going to display a coupon for some shoelaces at the very top of these results. And so, yeah, to kind of wrap it up, like, yeah, search on WordPress is hard because search necessarily has to understand and interpret data, to respond to intent. And because you can mix and match WordPress data any kind of way, building a search tool that handles all of that stuff nicely without a bunch of custom code can be difficult.
We like to think we’ve done a pretty good job of it, but definitely, it was not an easy experience.
DP: And Jared, this is, this is one of your gripes with search, right? Is a lack of support for tools like ACF?
JH: Yeah, I think so. As many folks know, ACF is part of the WP Engine family at this point, and, it is something we take super seriously. We know some of WordPress’, you know, biggest enthusiasts are ACF users, and the people that are most familiar with the issues with search tend to be more technical, you know, devs, folks that work for agencies, and, you know, with ACF being core to who WordPress is, or what WordPress is and who’s building all of WordPress, we just thought, search should work for these groups of very enthusiastic WordPress users.
And so we heard from all the ACF users and we know ourselves that if you’re investing in creating all these bespoke relationships between your data and default WordPress search is just not able to even support that in the least bit, that’s a huge issue, right? Just right out of the box. And so we looked around at other solutions out there as well and none of them, to Luke’s point, were really supporting ACF users and custom post types in a way that was seamless and really easy to use. It always either didn’t support ACF or, you know, it just, it required a lot of custom coding.
And so we wanted to make sure that we support that community because obviously, they’re incredibly important to WordPress as it stands today.
DP: Luke, you mentioned something, it was just a word, that you mentioned. You said the word language, and it just got me wondering. If large language models are the way, I guess, this feels like one of those things where every single problem we have on the show, at some point we just say, okay, an AI will fix this.
And I’m just kind of wondering, is that what y’all are planning on doing? Is a large language model or, you know, AI in some form, part of, you know, what you’re working on as a solution for search?
LP: Are you asking if we have been swept up in the AI hype-cycle?
DP: Mmm-hmm.
LP: Doc Pop, we absolutely have,100%. So, okay, I use the word “hype-cycle” really intentionally to kind of like, hey, let’s upfront acknowledge that we are in the process, collectively, as technologists, of separating the wheat from the chaff here, right?
What are the real core applications of this technology? And so, this is how we choose to think about AI at WP Engine, as a tool, right? We want to always be starting with problems and say, what are the tools at our disposal to tackle these problems? To tackle them faster, to tackle them cheaper, to tackle them in a way that is surprising and delighting for the folks that use our products. So, some of what you’ll see out there is just kind of like a GPT wrapper. Hey, write me a blog post about X. We don’t do that. I think there is enough. This is my personal opinion. I think there’s enough blog spam out there. There’s enough SEO-coded blog stuff out there and AI pushing out even more of that. I mean, that’s just, that’s just something that we’re not interested in, right, to go back to it. What, problem does that really solve for folks?
It’s cool. It’s interesting. It is an insane, um, breakthrough in technology, but I think for us, like, there’s not much value for folks that use WordPress in, in that, that we can see anyway. And so, to answer your question more concretely, what problems are we going to be applying AI to? Search!
So the things that LLMs are really good at are things like having conversations and things like summaries. And so we kind of have a three-pronged approach for, hey, how can we use what LLMs are really, really good at—and machine learning as well, so we’re kind of under the big AI umbrella—to solve problems for and around search?
So number one is tackling what we call semantic search. So you remember I said that example of, we want somebody to be able to type into a search bar, Hey, Luke’s website, I tend to overpronate when I run, what kind of shoes should I buy? Okay, that’s tough to deliver search results for that phrase. Large language models can convert that phrase to something that our search database, and we create what’s called a vector database, in order for an LLM to kind of talk to the, the content of the search index. LLMs are really good at translating that natural language into something that your search database is going to understand.
So that’s application one of, you can use plain language in your searches, on your website, and we kind of take care of all the stuff on the backend for you. The second one is recommendations. So I mentioned something called a vector database. A vector database is something that we take your search index, So that’s where all of your kind of searchable content lives, in a special search-configured database. We call it an index.
What we do is we take that index and we turn it into what’s called a vector database. So a vector database just kind of uses machine learning and a machine learning model to say how closely related do we think these kinds of data are? I’m way oversimplifying.
But what you can do after that is you can surface content and product recommendations, right? Because once we run our machine learning model, then shoelaces get really, really close to shoes in that vector database. And so we can automatically generate recommendations. Hey, you liked X. Oh, I bet you would like Y.
And so that’s kind of an AI-powered recommendations algorithm that is trained on the content of your website. So, Phase 2 for us is generating really, really smart recommendations. Phase 3. So, all this is about how large language models can talk to the content of your website using a vector database.
This kind of framework, by the way, is something called Retrieval Augmented Generation. It’s a RAG, and so you can look that up if you’re interested in that kind of thing. Um, but another kind of LLM implementation is a chatbot, right?
And so this is the biggest request we get from customers of, hey, can you train a chatbot on all the content on my website? Well, no, because there’s kind of a thing called a context window, and it’s too small to get all your content. But what we can use is a vector database and retrieval augmented generation to hook up your content to a chatbot. And so that’s Phase 3. Again, this is in the future for us, but the ability for us to magically turn your content into a vector database that hooks up really, really easily with these LLM chatbots.
And again, leveraging the strength of these LLMs, they’re really good at holding a conversation. Okay, let’s hook it up to the dataset of your website so we don’t have to train it, but we’re going to allow it to retrieve content and answer back at your customers and their questions in a really, really nice, natural conversational way.
DP: That makes sense. This is exactly the use case I think that you would want to have an LLM for on your site. So that’s interesting to hear about how that could help improve search. And we are going to take one more final break. When we come back, we’re going to continue our conversation with Luke and Jared about search, maybe along with some recommendations for users on some other search tools that we recommend. So stay tuned for more after the short break.
Welcome back to this episode of Press This, where we are talking about WordPress and search, a subject that many of us are passionate about. And during this episode, we’ve talked about issues that we have with search. Luke and Jared are working on a search tool and they’re very passionate about search. And I want to wrap up this conversation by just saying, hey, if you’re listening to this episode, here’s how you can improve search on your site, or here’s a tool that you can use. So Jared, uh, sorry, Luke, why don’t you tell us, um, what tools you would recommend for search?
LP: Yeah. So obviously we love Smart Search and we built it because we saw a gap in the way search offerings were treating WordPress, the WordPress community, and the WordPress ecosystem. All that being said, there are some folks out there who do search really, really well and work hard on a difficult problem. So on the kind of like a little bit less pricey side, where we’re typically going to see customers on is, Relevanssi and SearchWP. SearchWP is someone that, credit where credit is due, does really well with ACF. Folks tend to come to us once they get, like, a lot of searches, and a lot of posts. That’s the performance aspect that really kind of tends to bog down those Relevanssi and SearchWP customers at scale.
So you get to kind of the middle of the pack. That’s us, Smart Search. We handle performance really, really well. We have a customer that does like 10 million searches, and so the pricing reflects that. So we’re a little bit more expensive than your Relevanssi, SearchWP. I would call ElasticPress kind of middle of the pack with us as well.
And then on the higher end, you’re going to have Algolia, if you need all the bells and whistles. One, you’re going to pay for it. Two, Algolia is the big player in this space that’s going to offer you the most flexibility there.
DP: Awesome. That’s great. Jared, do you have any suggestions you want to add on top of that?
JH: Nope, Luke nailed it. I think I just reiterate that where we tend to, where we really wanted to hone in was ACF users. you know, As them being a part of our family, we wanted to build products for ACF users that make ACF work better with us. And so just now and in the future, um, we’ll continue to do that.
And that’s, that’s one of the areas where Smart Search tends to really be filling a gap from some of the other great competitors that we see out there.
DP: Awesome. Well, I really enjoyed chatting with both of y’all about search and WordPress, and we got to, we got to vent. We can let it all out now and talk about, other things. go pet our dogs and touch some grass for a minute. if people want to learn more about, what you’re working on, Jared, what’s a good way for them to follow you or where would you like to send them?
JH: I don’t do socials too much, but I would say find me on LinkedIn or just my emails. My first first and last name jared. hutchinson at WP Engine. If you have questions, suggestions, feedback on this, search, really anything, please feel free to reach out. We also have an Agency Partner Program at WP Engine and we have a Slack community there that’s really active. I’m always there chatting with agencies, so if you are an agency in the program, please DM me there on Slack and I will chat away with you. No problem.
DP: Awesome. And same question for you, Luke. How can people follow you online?
LP: Y’all can find me on LinkedIn and also on Twitter at LAPATTE31, L A P A T T E 3 1. It’s a little weird, but that is where I live on Twitter.
DP: Awesome. Well, I really appreciated having y’all on. It was really fun talking to you. Next episode, we are going to be talking to Amber Hines from Equalize Digital Accessibility about an accessibility plugin to help check your site to make sure it’s compliant, as well, we have an interview with the team from BuddyPress who are working on the Buddyverse. And we have an episode coming up with them talking about that. I’m super excited to get that out there.
Thanks for listening to Press This, a WordPress community podcast on WMR. You can follow my adventures on Twitter at @theTorqueMag, that’s @theTorqueMag, and you can subscribe to Press This on RedCircle, iTunes, Spotify, or your favorite podcasting app, as well as download directly from WMR.fm.
Again, I’m your host, Dr. Popular. I support the WordPress community through my role at WP Engine, and I love spotlighting members of that community each and every week on Press This.
