This weekend MLKSHK was pretty unstable. It started with a small issue of log files getting too big (since fixed) but then coincidentally we got slammed by some very popular forum hot-linking to 50 files at once AND RabbitMQ deciding to completely stop working. This meant SLOW buttons* and 500 errors due to too many files being kept open by the huge traffic spike.
Everything has been fixed at the time of this posting, but I there is a lingering problem with RabbitMQ I will have to figure out at some point. I would like to have someone work on MLKSHK more than just weekends, but we do not have enough money to pay someone for that work.
Please consider getting a paid account today. We have over 60,000 users now and a very small fraction actually pay to use the site.
*The like and save actions are queued so you don’t have to wait for them to complete. We send the request off and calculate the stuff later (measured in milliseconds). When RabbitMQ is offline we have to calculate likes in real-time and that is why the button feels so slow.
A guest post by tonyb
It just so happens that I love MLKSHK. I love reading it - I love contributing to it. It’s my go-to tool when looking for that special animated .GIF, and it’s the last-funny-thing I read before going to bed. In return, MLKSHK seems to love me right back - I get lots of likes and comments and followers. I get daily rewards that help me feel like a content provider - like I’m sharing an individual viewpoint that people value and enjoy.
From inside my fluffy happy MLKSHK cloud it seems like everybody should love it as much as I do. And so I was quite surprised when I spoke with a few friends that I knew as fellow users, and they told me they didn’t “get it”. They tried going on a couple of times and clicked around, but it just didn’t stick. They wandered off and didn’t use it anymore.
When I heard that, I made it my mission to find out why, and - if possible - help them see a different side of things. I sat down with my friends and tried to absorb the beginner’s perspective - coming on to this vast social network where you don’t know anybody, and being bombarded by a flood of images. How to make sense of it all? Over the course of a few of these talks, I’ve boiled down a couple of tricks that I’d like to share - a few easy things that anyone can do (even if you haven’t subscribed!) to make your MLKSHK experience more interesting.
Okay, so you just signed up for MLKSHK - now what? Is this supposed to be a social network? A place where you can park your vacation photos? And where are the funny pictures anyway? The good news is MKSHK can be all of those things - it’s kind of a Choose Your Own Adventure that responds to your choices. The problem is - if you never make any choices, you never get past the front door. So let’s get started and learn the lay of the land.
When you sign up and log in, you get your own personal page which shows you all the images from all the friends and shakes you follow. Now, if you just started, it’s going to look a bit bare since you’re not following anyone yet. So your first order of business is to find some things to follow and get the ball rolling. MLKSHK has provided you with a few neat tools to get you started: POPULAR FILES - If you were wondering where the funny pictures are, you found ’em! But it’s good to remember that the Popular page is also a great way to find interesting users. If a particular image strikes your fancy, click on that user’s icon and go to their page - maybe click through a few of their pages and like/save some of their images. If you find that person’s stuff interesting, then click the FOLLOW button at the top. Now everything that person posts will start showing up on your feed. You want to see more than 30 popular files? Just go to the MLKSHK user page and you can see all the stuff that’s ever been on the Popular page going back in time. Please note that the user who posted the file is now under the file instead of at the top - so you can still use this as a way of finding other cool users.
INCOMING - this is literally every image that gets posted to MLKSHK being updated in real time. The plus here is that you might find users that share an interest of yours even though their pictures might never make it to the Popular page. The minus here is that you may have to dig through a huge volume of images, and you might never see the interesting stuff that got posted 10 hours before.
FEATURED SHAKES - For me, this is the most useful tool of all - the encyclopedia of all the cool things that will get your feed looking more interesting fast. If you click the People link at the top, you’ll see a list of the most active and followed users. Click around - see who you find interesting. Or just take a chance and follow a few top contributors to get your feet wet. You can always un-follow them later in case their particular stream starts to bore you (follow in haste - unfollow at leisure).
And then there are the various GROUP SHAKES - here, rather than following a particular person, think of it as subscribing to a topic-specific image blog. Do you like cats?, or cars?, or men in gorilla suits holding scantily clad females? Well, chances are a lot of other people do too. One of the things I love most about MLKSHK is the huge variety of images it exposes me to - and group shakes are a great way to organize that experience and find other people with similar interests. Take the time to click through the categories on that page, and dig down into the various group shakes you can follow. You’ll be surprised what you find!
Everything you’ve done so far is to make MLKSHK more interesting to you - so what can you do to make YOU interesting to MLKSHK? I mean, reading MLKSHK is fun - but there’s a whole other side that opens up when you start posting images and interacting with other users. If you go up to the Choose a Shake button and hit Your Shake - you’ll see what the rest of the world sees when they click your user icon. Is it looking pretty empty? Well, let’s get some images up on there so that when people check you out they can see that you’re interesting, and maybe they’ll follow you - and like your images. A couple of easy ways to do this: NEW POST - got some awesome images kicking around on your hard drive? Sure you do! Once you upload them, you can change their title, post a description, and even flag them as Not Safe For Work. BROWSER EXTENSIONS - these are right on the front page under Cool Tools - little extensions for Safari, Firefox, and Chrome. So next time you’re doing a Google image search, or you see something funny on some other website, you can just right-click, grab that image on the fly, and save it to your shake.
If you’re just getting started and no one’s following you yet - it might seem like you’re whistling in the dark. Be patient - if you build it, they will come. There are no sure-fire tricks to getting followers. If you follow some veteran users, they will often reciprocate and follow you out of courtesy and curiosity - but there are no guarantees. The best thing to do is just to be active - post images to your shake - like, save, and comment on other people’s files. Eventually someone will notice your cool stuff on Incoming and you’ll start to see people liking and saving your images when you log in to your default page, and you’ll see what people are responding to. If that person is someone new to you - click on their user page and check them out. Maybe you’ve found a new person who shares your interests.
If you’ve found a group shake that you really enjoy, you can always request to become a participant of that shake - it’s the +Join This Shake button on the front of any group shake. Once the editor of that shake approves your request, that group shake will now pop up under the Choose A Shake button - and also on the pull-down menus whenever you save a new image to MLKSHK. If you want to post an image to two or more places, you can post it to your own shake - click on that image, and then click all the boxes for the various group shakes you want to list it in. Posting to group shakes is great because it can really feel like you’re having a meaningful exchange with other like-minded people
Okay, so now you’ve been using the site for a while - you’re posting cool stuff, you’ve got followers in the double digits, you’re getting new likes on your images every day, and you’ve even had a file pop up on the Popular page. You can sit back and relax now, right? Well, sure you can - but there’s so much more out there. New users are joining MLKSHK all the time - and there are probably thousands of current users that you haven’t even met yet. Keep checking out which users are saving and liking your files - that’s probably the best way of finding new people that might share your interests. Keep adding new people to your feed, and your MLKSHK experience will get richer and more interesting over time. Eventually, you may not even need Incoming, since your feed will become all the parts of Incoming that are interesting to YOU.
One of the things I hear from my friends is that they save an image to their shake, but then they can’t find it when they need it later. There’s some truth to that - MLKSHK is not Flickr or Picasa. It’s set up to provide you with a fresh interesting stream, but that doesn’t mean you can’t find what you need when you need it. Let’s say you’re working on a blog post, and you need an inline image of somebody doing a facepalm. Just go to Google Image Search and type in “site: mlkshk facepalm” - and bam! you’ve got every mlkshk page that has ever had the string “facepalm” in the title or comments of that file. Obviously that doesn’t help if your image was named “DSC5128149624.JPG” - so taking the time to name your images will pay off on the back end. The good news is that even if you don’t find that one image you saved - chances are you’ll find 5 images that other people saved that might work even better.
Okay, look - I don’t work for MLKSHK, so there’s no sales pitch here. All the awesome stuff I just told you about is available to any MLKSHK user free of charge. So why am I a subscriber? What do I get for $6 bucks a month that makes it worth my while? Well, take a look at my shake again - you see where it says “tonyb’s other shakes”? Those are all the group shakes that I created. To me, each one of those is a category of images that I’m personally interested in. As I cruise the internet, I am always on the lookout for more - and when I find them, I squirrel them away in my shakes. I then get likes on those images from other people who have joined those shakes - and happen to share my interests. Even better - I get to see images from those people showing up in my shakes - images I might have never found on my own. For me - this interaction between like-minded individuals is one of the most interesting things about the whole site. It’s more than just people throwing up funny images for all to see - it’s people who share something having a visual conversation with each other. We are the audience and we are the show. My name’s tonyb and I got no reason to lie to you.
Andy mentioned something about Imgur’s most popular files based on views, so I figured I could find something for MLKSHK.
So here are the top 20 files from this year based on number of likes received. Note that the first one was our full-page ad in Longshot Magazine that the members of MLKSHK paid for.
So I had a problem yesterday where EC2 restarted all of my servers. They all came back except one which I rely heavily on for serving pages.
Normally there would be two machines there, which is why when this sort of thing happened in the past nobody really noticed. It has been our plan to have two machines EVERYWHERE. (Everywhere except the database which was our last big outage a few months back. That was a MySQL RDS issue that, even if we had two machines I don’t know if we would have avoided that problem.)
Now when this happened in the past I would freak out and go fix it and nobody would notice. But this time it happened while I was asleep and had the vibration alerts off. (We were on vacation last week, oops.)
The reason why I did this is simple: I needed to save money. Running redundant servers is great for avoiding headaches but not so good for the MLKSHK bank account. I can start up new servers as needed, but to keep one in reserve is more than our paid subscriptions can cover.
Another thing you might have noticed and I wanted to point out: this is the first post I’ve made here where I use the word “I” instead of “we”. That is because it is just me running MLKSHK now. This is a little bit of an uncharted territory for me, because friends have said I shouldn’t point this out, that the site is fine without anyone knowing what’s going on behind the scenes, but I think that faceless “we” is what leads people to think we’re some kind of funded startup.
We aren’t. We (my wife and I) built MLKSHK with our personal savings. But now we are going back into consulting because we couldn’t make the numbers work out right to keep spending all of our time on MLKSHK.
But I also want to point out MLKSHK isn’t going anywhere. In fact we have more people visiting the site than ever before. Without that redundant server we are well under the cost to keep the rest of them online, and there are further changes I can make.
So, please, if you have a free account, consider upgrading. If you enjoy the service and want to see your friends on it, please invite them to check it out. Every day is a FREAKING JOY to visit the site for me, and I hope all of you.
I mentioned on Twitter the other night about a hiccup we had on our server and someone asked if I could give a high-level view of what makes MLKSHK go.
First, we use the Tornado framework. It is fast, simple to learn, and best of all: non-blocking. The architecture for the originally planned MLKSHK called for a bit more use of this lovely asynchronicity, so we chose Tornado to do this.
The other reason we chose Tornado is that it’s actually enjoyable to read the source code. It’s small enough that if you are stuck on something, it’s easy to pop over to the source and see what’s going on. I feel this way about many Python projects, but never more with Tornado.
Using Tornado though, means you have to build or find some of the components that are common in bigger frameworks–specifically ORMs. We rolled our own that is just the right amount of thin-layer on top of MySQL. It’s called FlyingCow and does a great job inside of Tornado.
Nginx does two huge things for us: first, it’s a proxy for file downloads from S3. We use the X-Accel-Redirect header to tell nginx to proxy the file from another location. Second, it handles file uploads using the nginx-upload-module. If you are using nginx and are doing your own file uploads stop right now and use this module.
The thing that hiccuped the other night was RabbitMQ. This has been happening frequently, but I haven’t figured out exactly why. My first guess is some sort of memory issue but I haven’t been able to figure it out from the logs. Either way: it gets stuck. In front of that is Celery, we use queuing to handle some of the more lengthy processes like favoring files and saving to your shake so you get an immediate feedback on the front-end. Plus if we are getting hammered with requests you won’t notice it as much.
We use HAProxy to balance the load across multiple machines. At any time we can put new machines in rotation to lighten the load. Airbrake to notify us of errors. Postmark to send mail. Github, of course. And EC2 to host it all.
…And How You Can Help
We had some downtime this morning after a maintenance process got out of hand and choked the database server. Because of this the front-end servers couldn’t serve content and backed up a lot of processes.
I was able to get the site back to normal very quickly, but there exists a lingering problem I haven’t been able to track down. Likes are intermittently not registering. If you could be on the lookout for a bug where liking a file doesn’t save the like on the image’s page that would really help us. If you see it happen, could you mail us* the page where the error occurred?
We’ve done a few things to help prevent this issue in the future, but I’m afraid the next step is ordering a bigger machine to handle the data. Hopefully recent changes to our architecture will save us enough money to spend on upgrading our DB.
*email address is at the bottom of every page on the site
Hey! Another one. Let’s see:
DNS outage this week: bad. New DNS server coming: good! That was unpleasant, but it was partly our service’s mistake, partly my own dumb mistake for setting the TTLs (time to live) so low. You can think of TTLs as a note to your browser* when it should update where to find our IP address. Normally you use a cached version and ask for an update every so often. We were doing some DNS work and I set it to such a short time that the second our service had a hiccup, nobody could find the site. On the plus side, with such low TTLs we were back quickly.
Spam is still an issue. I thought I had them licked but they keep figuring things out. Onward!
What is next: we have many things planned but are always interested in hearing what you have to say. Please let us know what’s important to you and what isn’t working.
We‘re adding people at the fastest rate since our launch. That means you guys are telling friends about us and sharing images on Twitter or Facebook. Thank you so much for that. We love you guys.
* I know. Easier to explain this way.
Our DNS provider has gone offline. We can‘t even reach their admin console.
You can hit MLKSHK with this IP address:
We‘re very sorry for this. I don‘t have an ETA yet but will keep you updated.
All systems are back to normal. I updated Twitter and my personal Twitter account but forgot to swing back here. We’d love to hear of your beloved DNS providers in the comments if you have the chance. Going to do some research this week…
It’s been a while since we’ve posted, for that we apologize. We’ve been extremely busy the past month and a half as we had to return to consulting to pay some of the bills.
MLKSHK saw a tremendous amount of growth and traffic last month. Over 1 million unique visitors hit the site as the sheer size of our database of images means it’s very viral-friendly and usually there are a few images getting a ton of traffic.
We have pushed a few new features we’d like to tell you more about. Some you might have noticed, some you might not have.
First, we have become very attractive to spammers for obvious reasons. Because of that we’ve been spending time trying to knock them down. It’s not easy, especially with such a small team as ours, but I think we are doing a pretty good job of keeping the comments clean.
If you see something please report it. Either through the email address or hit us on Twitter with a link: @mlkshk.
Next is something we have been meaning to do for a while with the bookmarks:
You can now jump back to a previous bookmark and begin reading forwards. We’re still working out the kinks, and thinking about how to better present the experience, but we find it INCREDIBLY useful and we hope you will too.
Note that it won’t work for your old bookmarks, only new ones as they are created from this Saturday forwards.