Welcome to Press This, the WordPress community podcast from WMR. Each episode features guests from around the community and discussions of the largest issues facing WordPress developers. The following is a transcription of the original recording.
Powered by RedCircle
Doc Pop: You’re listening to Press This, a WordPress community podcast on WMR. Each week, we spotlight members of the WordPress community. I’m your host, Doc Pop, I support the WordPress community through my role at WP Engine and my contributions over on TorqueMag.io. You can subscribe to Press This on Red Circle, iTunes, Spotify, or you can download episodes directly at wmr.fm.
Pumpkin spice lattes are back in season and 12-foot-tall skeletons are back in stock at Home Depot, which means Halloween is here again. Now, Halloween’s origin goes back to ancient Celtic festivals when people would light bonfires and wear costumes to ward off ghosts.
Going with that spirit, we thought we’d spend today’s episode of Press This telling spooky stories of clients from hell or installs gone wrong. So come sit by the fire with me as we hear from our first guest, Chris Wiegman, an Engineering Manager at WP Engine, as he tells a frightful tale. Chris, can you set the mood for us here? Before you get into your tale, tell us about when it’s happening and where you are in life. Kind of set the mood for the story.
Chris Wiegman: This is an old tale. It goes back to a state far away and a job many removed from what I’m doing now. It was one of my early jobs in the WordPress industry. This would have been winter to spring of 2014. So almost nine years ago that this occurred.
DP: 2014. And you were telling me earlier that you had a successful plugin. Can you tell us about that plugin that you’re going to be telling the story about?
CW: Sure, the story is also about one of the early plugin successes. It’s a plugin now called iThemes Security, and I had sold it to iThemes as Better WP Security. When I sold it, we knew we had over 200,000 active users, remember this was nine years ago.
And that was before they even had download counts available and all the horror stories of—that’s been removed now. This was before that was even a thing in the first place. So we knew we had a lot of users, we knew we had a very successful plugin, and we were releasing the first version of it as a rebrand from Better WP Security to iThemes Security.
DP: You kind of actually got me thinking here, you’re talking about before the days of tracking active installs. Do you know roughly how many downloads you had, or how back then would you have known how many active users you had?
CW: The best we had to go by was active downloads and a lot of educated guesses. At the time, I had put it in Google Analytics so that cleared up a lot of things, but before the Google Analytics part, the only way we knew was how many downloads, and then we could guesstimate based on that how many active users we thought there were.
DP: Okay, so you had a very popular security plugin, a WP security plugin. You had a lot of active installs and probably a lot of active users. You were saying it’s kind of one of the early WordPress plugin success stories. And you had just sold it, so things are going well for you. This is a success story so far in your horror story, right?
CW: So far. Yeah, things went well. We had a support person now to help me full-time with it. I had time, being part of a company now, that I could dedicate full time to developing it so we could actually do things like a beta program and get people testing it, and make sure everything was working and really spend time to make sure that as we prepared that first release, that everything should have been right, should have been working.
DP: So things are going well. Why don’t you get into the meat of the story here. Tell us what happened?
CW: Sure. Well, what happened is that the plugin had two features. The first was something that was wildly considered part of security 10 years ago in WordPress, which is, I mean it’s not security at all, in reality, especially we know that these days, and that’s hiding the admin area or hiding WP login.
Normally you go to the URL and your login form pops up. We used to like to say—and I’ll admit now that it was probably more marketing even then than it was security— that if you hid those, somehow the site would be safer. It’s kind of like taking the front door of your house and putting it along the side of your house and claiming that your house is more secure. It didn’t work that way. But it was a very complex feature to make this happen, to make people feel better about it.
And the second feature that we had that broke on this release was something called “away mode,” which is basically, if you were in a nine-to-five office and you didn’t want people getting into it at all outside of work hours, say for a school or a doctor’s office or something like that, this feature would set the times that that form was available at all. So you couldn’t log into the site. You couldn’t get into the backend except for the hours specified by the site admin.
And both of these features were really rewritten pretty heavily for this. I think the number was the 4.0 release that we came out with, around the beginning of March. So this was four months from the time I had started and sold it to being able to finally work on it full time. You know, it had always been a side job for me. I had a full time job at other companies and schools. This was the first time I’d been able to work on the plugin full time.
So we started a beta program where we rewrote these features and we released them on, I want to say right around March 1 give or take. So for two or three days after that release, everything looked okay. The feedback seemed positive, we had never heard any problems with the beta testers. We had a few 100 people beta testing this thing, and everything seemed good. And then the problems started to roll in. People couldn’t get to their site. I can’t remember now if it was just the dashboard, I think in some cases the whole site went down and we couldn’t figure out why.
Okay, you must be doing something wrong. Things changed enough on this and maybe your configuration was wrong. And we kind of went with that for a day or so. And then it exploded. We didn’t bring down one or two sites with a bug, we did find the bug and we patched the bug within about a week from the initial release. We brought down somewhere between 10 and 20,000 sites by estimate with this bug, based on usage numbers and how many people we knew were using the feature and this and that.
The combination of the away mode and the hide backend simply left things in loops. You couldn’t log in, you couldn’t get to things, the site was effectively dead. And you couldn’t just log in and disable the plugin. You had to get into hosting. Now, if you’ve ever dealt with hosts and WordPress users, sometimes it’s really easy. It’s easy to tell somebody hey, login and disable the plugin and that’ll fix your problem. But what if you can’t log in? How do you get to that plugin to disable it? So we wound up with, you know, a couple hundred one-star reviews all within a couple of days. It just exploded and it was not a good situation for anybody.
DP: Wow. So you think somewhere between 10 to 20,000 sites that were using this plugin, and got this beautiful update that was four months in the making, and had a beta program, and was well-tested, you had a team behind it. But it didn’t instantly bring these sites down, but kind of a rolling series of crashes. It seemed at first like user errors, and very quickly became, “Nope, this is something we did.” So how did you react, what was the next step for you, after you know, hundreds of negative reviews and thousands of sites down?
CW: Well there’s only so much you can do. Our goal then was to A) fix the problem. Some folks came back, many, over time, came back, some folks would never touch the product again and it’s understandable. We broke trust with it.
And then the second thing was making sure that we had even improved beyond that. Some of the feature testing and things like that for future releases. I was only with the plugin for 14 months total with iThemes before I left working on that plugin entirely. We instituted quite a number of programs, quite a number of tests, quite a number of procedures to make sure something like that couldn’t happen again. And to my knowledge it never has with that type of plugin or with that particular plugin.
DP: That must have been stressful for you. Do you feel like you’ve learned anything from this experience? Is there anything you’ve changed since then because of this experience?
CW: I haven’t done a commercial plugin like that since. One of the lessons I learned is that even a beta program, you got to get the right data back in the beta program. Could we have discovered this? Well it was pretty clear that with the right combination of settings, yeah, we probably could have discovered it in beta. But instead we had just opened a beta and said, “Here try it. If you see anything, let us know,” without any kind of direction. Instead of ”try this specifically,” or “try that specifically.” You combine that with everything else going on. If there was a weakness it was that lack of direction, lack of inspecting each feature in the beta and just looking at everything from the big picture rather than specific details.
DP: So if that tale that Chris just told you of 20,000 sites going wrong after what sounded like a successful launch of a new plugin, then stay tuned after this break for another horror story. We’ll be right back.
DP: Welcome back to a Halloween edition of Press This, a WordPress community podcast. On this episode we’re telling frightful tales of plugins gone wrong and other WordPress horror stories. I’m your host Doc Pop and right now I’m talking to Derek Ashauer, a web designer and developer who also makes WordPress plugins. Derek, I hear that you have a spooky tale for us. Can you set the scene?
Derek Ashauer: Yeah, so this is really early in my career, a long time ago. I was still working full time at a normal company making and building websites but I was doing some freelance work on the side. I had helped a small concert venue build a custom ticketing system because they really hated Ticketmaster. They were an indie kind of venue so they wanted to do anything to avoid those big corporate companies. But I built this pretty good ticket system, I thought at least. And they were going to have a huge concert back in the day when Blink-182 was really popular. They were going to have them at their venue and they were going to sell tickets for $1 a-piece. So this thing’s gonna get absolutely slammed overnight when they release the tickets.
So we set it up, did all kinds of testing and thought it was working great. And then come the morning that we’re supposed to release it. I think it was like a Monday at 10:00 in the morning. There were some rules, some basic things that we had in place, like you couldn’t buy more than eight tickets to try and give as many people the opportunity to get tickets and stuff like that. Again, early in my career, so I didn’t do the best at checking on things. But the venue itself could hold about 1,000 people. So we had a limit that once it hits 1,000 tickets to basically stop selling.
We released at 10 o’clock, and I’m at my normal day job just doing my thing. I kind of checked it, to make sure the site was at least loading and stuff like that but not really too concerned. A few minutes go by and everything seems to be going great, going on. And then, suddenly I started getting text messages. And then I get a phone call. And then I get another text message and I’m in the middle of my job just doing my normal thing so I couldn’t really just easily take these.
It turns out that I forgot to do the little query check to check the max tickets sold. And suddenly it was going to 1,000, 1,050, 1,100, 1,200. I think it got upwards of about 1,600 tickets sold before I finally was able to log into the server and just basically pull the plug. And so obviously, the owners of the venue were panicking and completely freaked out that they had a thousand-seat venue and had sold about 1,600 tickets, and so they were just obviously panicked. And I’m in the middle of my workday panicked. How am I gonna solve this? How am I gonna do this? I gotta do my normal work stuff, and deal with this freelance thing. It was a total disaster at that moment.
Thankfully, things did end up working out perfectly fine. What was interesting is that another unfortunate thing that I didn’t check was, one way people got around the max tickets was they would just buy multiple times but use the same email address. Again, this was very early in my career. I wasn’t very good at figuring out how to handle possible situations that people would try to work around. So they went through and they checked all the orders, and realized one person with the same email address ordered 24 tickets, so they reached out to them, refunded them, and did that as much as they could. And they got it down to about 11-1,200 tickets. This is so long ago, I don’t remember the exact numbers. But they got it down to that about that many.
And then the day of the event happens and they’re still a little worried about being able to fit everybody. I think about only 600 people ended up showing up. The reason being it was just $1, so a lot of people bought the tickets just in case and then a lot of people couldn’t show up. And so they never ended up having a capacity issue. It all ended up working out well.
But it was some stressful times when all those tickets were processing and getting paid and doing all that kind of stuff. And thankfully, the client was very happy and understandable in the end, they weren’t angry at me. They ended up using that exact ticket system once I patched that little thing. And they ended up using that ticket system that I had made for about 10 to 12 years. And so yeah, they were pretty happy and we got it all sorted out. And even for me, the client did all the legwork of reaching out to all those purchases and doing all that kind of stuff. So I just kind of had to turn the server off and then fix the little patch didn’t have to deal with too many of the consequences, thankfully. But it was a very stressful couple of hours while we were trying to figure out what happened and what was going on there.
DP: That was a roller coaster, Derek. You were setting up this scenario and I’m kind of getting little hints of when it happened. You know, Blink-182 are kind of popular. I’m assuming you needed a custom plugin because there weren’t very good options like there are now.
DA: It was 2005. A long, long time ago. Somewhere around there, yeah.
DP: You were building a custom plugin. So okay, so the height of Blink-182’s popularity, and tickets are $1. That’s insane. So obviously there’s gonna be a lot of demand. This whole roller coaster of like, “Oh no, we sold too many.” I thought you were going to tell me you sold by tens of thousands more. I feel very lucky you only oversold by 600 tickets because this could have been much worse. And then the scalpers, boy it worked out. Especially because the client could have put all of this on you to like reach out, and do tech support, and cancel these tickets. Man this was a roller coaster.
DA: Yeah it was. This was my first real large development thing, the biggest thing I’ve ever developed was this. So I just had no idea of how things could go wrong, how badly things go wrong, what to even check and it was just a very good learning experience, that’s for sure. I had a good relationship with the client, so they were pretty happy, because honestly, it was early in my thing, I was charging next to nothing. So it wasn’t like I charged them $50,000 for this thing, and then all of a sudden it didn’t work. I was getting paid honestly, on a per-ticket basis. I got 10 cents a ticket at the time, that they sold through their thing, and me being in my early 20s and making a couple extra thousands dollars a month. That was phenomenal. It was wonderful. So it was a great situation. But yeah, like I said, they kept using it for over a decade, the exact same system.
DP: So you built this ticket system for a pretty big event. And that event, as we said, kind of spiraled out of control. But it sounds like the two issues were having some way to prevent scalpers from at least using the same email.
DA: Yeah, exactly. I didn’t even do that, because there were no user accounts in the system. It was pretty straightforward. It’s just a one time guest checkout. So it didn’t even check email addresses or anything. And it did keep track every time a ticket sold. It kept track of a total. It’s just when people went to the page, it forgot to check how many tickets have been sold and have we passed that number, and to stop it from selling more.
DP: So those two things got fixed and this worked for 10 years pretty much kind of running itself?
DA: Yeah, I never touched it really ever after that. It just kind of kept cruising along until they finally got big enough to where they kind of had to do some business merger, like one of those other music companies, I forgot what it is. They kind of got bought up basically, and so then they were like, no, we have to use Ticketmaster or some other thing like and so they eventually were forced to abandon it for business reasons.
DP: They probably got acquired by Clear Channel or something.
DA: Yes, that’s what it is, Clear Channel. Yeah, it was something along those lines.
DP: So just kind of looking back. What is the one bit of advice you’d give to someone tackling a project similar to this, based on your experience. What is the one thing you’d warn them about?
DA: I mean, it’s obviously testing. That’s a big deal, is just testing your thing as much as possible and in as many scenarios. I mean, I still do my own plugins now and I actually just got a request for one, just this morning, actually, where I responded back, “I never even considered someone doing that. Ever.”
I have a confetti plugin, and he was like, “I put my confetti twice on the page. Once the page loads and as the user scrolls down, then it’ll go again.” And I never considered anyone doing confetti twice on one page. And so you know, you can test as much as you want, but sometimes you’ll run into those scenarios that you don’t think of, but you still have to do as much testing as possible.
DP: Derek Ashauer, I really appreciate your time. You’re listening to Press This. We’re going to take a quick break and when we come back we’ll have one final Halloween story to give you chills. So stay tuned.
DP: Welcome back to Press This the WordPress community podcast on WMR. This is a special Halloween story. Earlier we heard from Chris Weigman and I thought I’d have Chris come back and listen to the only WordPress horror story I have.
Chris, are you familiar with Midjourney and text to text image generators?
CW: Like Dall-e and things? Open AI and stuff like that?
DP: Yeah Dall-e. I was using them and kind of experimenting with different things and as someone who yo-yos a lot, the very first thing I tried was the yo-yo emoji. And the yo-yo emoji just really didn’t get any great results in there. It didn’t get anything that looked like a yo-yo for instance and the word yo-yo also didn’t get me stuff in text-to-image generators. But it really got me interested because I kept getting very consistent results. Whenever I used the yo-yo emoji, I would get this really cool-looking pink and blue scene with three mountain peaks in the background and a figure in the foreground. And this is supposed to be kind of random, and I kept getting very different images that had pink and blue pastel colors and figures and foreground and things like that. So I really started diving into why is this emoji giving me this and I spent hours going through different combinations of emojis. What does this emoji do? What happens when I do two yo-yo emojis?
And I wrote this massive blog post. This was gonna crack the case wide open on like weird stuff that happens in, you know, Dall-e and Midjourney around why does this emoji give me this image? And why do other emojis actually give me, you know, a pretzel will give me things that look like baked goods or coffee will give me things that look like a coffee shop. But the yo-yo emoji keeps giving me this strange scene.
And so after I wrote this massive blog post, I mean it was hours of research and documenting and taking notes. And then the writing, and I hate writing, it’s like pulling teeth, and I hit publish and go to sleep. It’s Sunday night and I spent all Sunday working on researching this post.
Monday morning people are just like, “All I see are squares when I go to your site, Doc. I see you say the square emoji gives me this result but the square emoji gives me this result.” I went and I checked it. You know the dashboard looked great on the backend, like on my side and the Gutenberg editor. It looked beautiful. On the frontend it was all squares. And all of that work was just absolutely shot. You know why, Chris?
CW: Why would that be? Ghosts in the machine? Gremlins?
DP: My WordPress site is so old, the database did not support emoji. Like at all. It was like 15 years old. If I would have installed something in the last eight years, it would have still been old but it would have supported emoji at some level.
My database from my WordPress website did not—and if you don’t really know what you’re doing, the one thing you don’t want to do is poke around in your WordPress database. That’s what I needed, to update my WordPress database, so Chris, that’s my horror story. I went looking for a plugin to just easily convert to something that supports emoji. Any of the databases that could do it. And now I’m going to have to hire someone just to update the database so that I can get this post that I spent 10 hours on to actually show up on my site in a logical way.
CW: That’ll do it. Old technology is a zombie waiting around to cause your problems, right?
DP: Yeah, you know, and it just, it taught me a lot too. Like, I can go into my portal on my hosting and I can, with a click, update my PHP. I can do all this other stuff. But yeah, that database, nope, you gotta know what you need. There’s no easy fix for that. And I think there maybe might have been, as those were rolling out, but I kind of missed the wave, like even those things that fixed the database, you know, to kind of update them are at this point, they’re even old technology, so that’s my WordPress horror story. And Chris it didn’t bring down 20,000 websites, but to be honest, it was a bummer and it still gives me shivers to see that blog post and think of what it could have been.
But that’s it for our Halloween episode of Press This, the WordPress community podcast on WMR. I want to thank all my guests for joining me today. Chris, thank you so much for joining me. You can follow my adventures with Torque magazine over on Twitter @thetorquemag or you can go to torquemag.io where we contribute tutorials and videos and interviews like this every day. So check out torquemag.io or follow us on Twitter. You can subscribe to Press This on Red Circle, iTunes, Spotify, or you can download it directly at wmr.fm.
We’re a weekly podcast, next week we’re going to have Fran Agulto, telling us how to overcome your fear of going headless with WordPress. We’re going to talk about the pros and cons of headless and if you’re worried about making that jump into headless, you’re worried about learning JavaScript, or whatever you need to do to kind of make that jump, Fran’s gonna have some great advice for you, so stay tuned for that episode. I’m your host, Doctor Popular. I support the WordPress community through my role here at WP Engine and Torque Magazine and I love to spotlight members of that community every week on Press This.
