May 14th, 2012 @ // No Comments
Hello, I am Anthony Skinner, the CTO of MozLand!
Many of you were affected by several SEOmoz tool issues that happened last week, unfortunately all colliding into one colossal day of craziness on May 3rd. We want to first apologize for any inconveniences or problems that these issues caused you.
The good news is that our awesome engineers fixed these problems quickly, but we want to share an update of what happened, how we fixed it, and what we’re doing to prevent “colossal days of craziness” from ever happening again. So, here’s the inside scoop (y’all know we like that whole transparency thing ‘n stuff ).
So, down to the nitty gritty of what happened last week and where we are now….
Rankings – Rankings were delayed by a couple of days for all customers due to some intermittent outages in our database. This delay caused custom reports without rankings data.
Fix: After trying it the hard way, we had a eureka moment (in the shower, no less) and promoted our back-up disks to primary, resolving the problem almost immediately.
Why it won’t happen again: We had planned for SSD failures, but did not expect to see a full cluster failure at one time. Going forward, we’ll be looking at making sure we’re using SSDs appropriately, and, when we do use SSDs, having more robust failover plans in place. We’re also changing the way custom reports are built to speed up the process, and enhancing custom reports to wait on dependencies.
Slow Open Site Explorer CSV Reports, and Mozscape API calls were failing – The Mozscape API was running noticeably slower and reports weren’t finishing. We found two export jobs that were continually requeuing themselves, severely backing up the CSV reports queue.
Fix: We fixed the condition causing the queueing and made some adjustments to the load balancing on the servers.
Why it won’t happen again: To prevent the queues backing up in the future, we’ve added a hook to prevent failed jobs from re-queuing. Monitoring and alarms have also been added to notify our ops team if these queues start backing up.
Campaign Setup and Custom Crawl – Users were running into an error message when trying to create new campaigns, and some users were seeing a dramatic reduction in the number of pages crawled.
Fix: With some creative ops magic, our engineers were able to configure the proper permissions and get campaign creation working again. Truncated crawls were caused by a race condition. We also made the transition between finalizing the crawling of a campaign and scheduling the next crawl smoother, which resolved this race condition. Affected campaigns were re-crawled so users could receive a full weekly crawl.
Why it won’t happen again: We’re working to do better testing at scale and to create more defined unit tests to catch these types of race conditions that don’t appear in small scale testing. We’re also working on better monitoring around the campaign crawl service and decoupling campaign creation from the custom crawl service so back end crawler problems will not have such a dramatic affect on the usability of the rest of the SEOmoz PRO app.
Delay in SEOmoz PRO Web App picking up the new index – Our latest index update wasn’t reflected in the SEOmoz PRO web app right away.
Fix: We redeployed an old endpoint in our API that we had been using for campaigns to pick up the new index metrics. We also updated the PRO software to use the new endpoints that Mozscape API now supports.
Why it won’t happen again: We updated our release procedures, and also updated the PRO app to use a new Mozscape API endpoint that publicizes the index launch date. This improvement will mean much smoother updates to Mozscape API campaign metrics in the future.
Social – PRO users trying to connect their Facebook accounts were receiving an error message. We were getting odd data back from the Facebook API indicating users’ authentication data expired – like 25 years ago .
Fix: We’ve updated the Facebook connection to return the correct time format.
Why it won’t happen again: To be honest, we’re not sure it won’t… We’ll try to stay on top of changes in Facebook and update our app before the changes affect our users.
We’re also going to be putting some of the new funding (read the memenouncement here) towards making sure things like this do not happen again. We’re investing in infrastructure improvements (blog post to come) to both help keep things running smoothly, and bring you new features and improve stability all around. We’re also hiring… if you’re a brilliant, motivated SEO-lover, apply here.
Again, many apologies for the inconvenience this caused all of you. We’ve learned a lot in this process and will keep doing our darnedest to keep things running smoothly.
May 14th, 2012 @ // No Comments
(Note, if you have thoughts about this video you’d do well to read my follow-up comment about the difference between reports and reportings.)
Although there are many ways of affecting change, project management is a crucial part of it. Below I’m going to outline a bunch of tools, tips, and tricks that we’ve discovered and implemented over the years at Distilled to get better at project management:
Personally I’m not a fan of clutter, either physical or virtual, and so I love technology that gets out of the way while you get on with getting shit done. I’ve tried lots of different project management solutions, and Trello is the first one I’ve fallen in love with. For those that saw the whiteboard friday I did with Jamie about a year ago it models the real life post-it note system very well:
I’m going to let Will explain why he likes it so much:
And I’m going to let Paddy break down the details in his blog post Using Trello to Manage SEO Projects.
A lot of you will already be familiar with Google Docs. Of course. But only recently have I come to realize the extreme power behind the collaboration elements. I’ve always hated track changes in Word and finally Google Docs has something better to offer. This video, although cutsey, actually demonstrates the power of real time collaboration:
In particular, two features that are really making me excited are in-line comments (with easy replies and notifications) and revision history (which allows you to see when, how, and who edited a document).
We use Google Docs extensively within Distilled to craft and send around documents even if ultimately we deliver the final report as a .PDF or some other format. After all, some large corporations still like the smell of .PDFs in the morning….
I can’t explain how much of a life changing experience the inbox zero methodology is. For the modern day information worker, inbox zero is fundamental to happiness and productivity. If you’re not using the inbox zero system then please trust me when I say it’ll change your life. Here’s Merlin Mann talking about the original system at Google:
When new employees start at Distilled, we coach them in the ancient ways of Inbox Zero. Although it’s a personal revelation for many (myself included), the real power comes when you have an entire organization that is GOOD WITH EMAIL. Having seen a peek inside companies that are not so efficient with email the difference is night and day.
Life is organized chaos. Sometimes not so organized either. Project management is similar in that it’s often more chaos than management. There’s only so much you can really and truly work to get things done without being in the thick of it.
So, where possible, our consultants aim to spend some time on-site with our clients. The increase in results is striking. Not only are we better able to communicate our ideas, but we are also better placed to understand how the client’s business works – not just the business model and mechanics, but communication, project management, hopes, and fears.
The best substitute for this if you’re not able to get face-to-face with the client is to at least communicate often with many different points of contact within the client’s organization. This improves the chances that you’ll understand the real needs of the client as well as ensure that as many people as possible like you which is important for getting things done!
We have various memes within Distilled; you can read more about them in a post I wrote for Dharmesh a little while back called Startup Culture Memes – Do You Have A Duck Of Awesomeness. One of the ones I’m most proud of is the mantra “communication solves all problems”. I’m constantly amazed at the ability to solve problems by communicating effectively. Either talking to other members of the team or talking directly with the client – just having some real interaction (face to face or on the phone ideally) and explaining the situation clearly solves 99.9% of all problems.
This mantra has infiltrated all parts of Distilled, but I see two key ways that this affects project management on every project.
At the start of any project, we have a kick-off meeting which has two clear outcomes; the first is a top to bottom understanding of the client’s business, and the second is a detailed understanding of what the project is going to look like. Mark wrote up our project kick-off process in a little bit more detail here: How To Kick Start SEO Projects.
Secondly, I drill into people here that it’s okay to miss deadlines. Really. It is. Do people really care if you deliver something on Monday morning instead of Friday afternoon? The answer is that yes, they care very much if you don’t let them know. If you let them know that you will deliver it Monday instead of Friday, then in 99% of cases, they could care less. Why is this so powerful? Because a single missed deadline without communication tarnishes your perception in the client’s eyes. So long as the communication is strong, the actual dates rarely matter.
PPT? As a project management tool? Well yes. Let me explain – there are broadly speaking three types of work that you do when you’re consulting and there are three different tools you use for these tasks as follows:
Although this seems like a no-brainer, it’s actually a very powerful mental model. Want to take a guess where setting the project vision and goals comes in? Yep – PPT. Although you won’t keep track of a project in PowerPoint, you should be crafting and creating the vision and goals for the project in PPT. Without strong vision and goals, projects will fail.
So persuading a consultant to put together PPTs at the start of projects is a powerful tool to ensure we have a clear idea of where we’re going, and importantly, the client is on board.
As part of our monthly reporting communication, we provide a letter from Will to our clients. This is a value add that allows our clients to keep abreast of industry news and changes. I’ve included a sample of the letter (and supporting links) for April here:
Why is this important? Well not all of our clients are SEO junkies like us. And they like to be kept abreast of the latest happenings in the industry.
How is this a project management tool? You might think it’s tenuous, but actually it’s crucially important. Running SEO projects on the shifting sands of Google’s algorithms means we have to keep on our toes and be prepared to potentially shift our strategy at a moment’s notice. So communicating these changes to our clients allows us to be on the same page when we start talking about pandas and penguins….
It’s important to note that what works for us may not work for you. Hopefully this has been helpful for you to take a peek at how we manage projects and communication. I’d love to hear what you guys have to offer in the comments!