Our Apologies to @BreakingNews

Some of you may have seen the following tweet which unintentionally ended up in the @BreakingNews Twitter stream:



I have to take full responsibility for this mistake (including the typo in the tweet). This tweet was intended for my personal Twitter account which feeds into my Facebook account.

The Adjix system has several code paths that support features dedicated to special Twitter users such as @GuyKawasaki and @BreakingNews. While testing some new features, I received a text message for free food. Just show the cashier your cell phone coupon - no purchase necessary. At the restaurant, I noticed customer after customer showing their cell phones to the cashier. When I returned to work, I tweeted this out from a development machine which inserted the tweet into the @BreakingNews Twitter stream instead of mine.

I didn't even notice the error until about 15 minutes later when Michael van Poppel, the founder of @BreakingNews, contacted me. There wasn't much I could do except apologize.

Unfortunately, mistakes like this can't always be avoided especially when rolling out new features. I worked for Apple Inc. for more than eight years, including a stint at the Apple online store. Once, the online store accidentally released an upcoming feature set to the public too early - about a week before Apple's World Wide Developer Conference (WWDC) in 2003. During the WWDC keynote, Steve Jobs referred to this as "premature specification". Accidents happen.

Cheers,
Joe Moreno
President
Adjix

Kobayashi Maru - Dealing with downtime

One of the worst things that can happen to an online business is downtime. It doesn't matter if the servers crashed, the network connection "broke", or the data center was compromised; your users can't access your service which is bad.

Theory vs. Practice
In theory, there is no difference between theory and practice - in practice, there is.

Ideally, a website should never suffer any downtime. In reality, downtime can't be avoided and it will happen whether planned or unplanned. Unlike brick and motor structures which are designed to withstand storms and earthquakes, online businesses are very fragile with many single points of failure such as DNS, loss of power, failed hardware, the easily cut T-1 or OC-3 cable, etc.

Implementing a contingency plan isn't an easy task and your plans will only work well when systems fail as you expected them to fail. You not only have to figure out what you're going to do under certain circumstances, but you have to rehearse it on a regular basis - this is what the military refers to as "exercise training". Not exercising such as pull-ups or push-up, but rather wargaming the scenarios which are most likely to occur and then actually executing the planned response.

Contingency Planning At Adjix
Adjix is a small company with limited resources, so we've implemented a simple solution to keep links running if our servers go down. We do this by relying on Amazon's web servers.

Every time our users create a shortened Adjix or ad.vu link, we implement it as a meta-refresh web page instead of the industry's more common HTML redirect (sometimes referred to as a 301 or 302 redirect). Should our servers go down, we can make a quick DNS change which takes about five minutes to propagate throughout the Internet. We believe broken links are a bad thing and although we may not be able to capture detailed link click data when this happens, our links will continue to work. We've tested this plan, at Adjix, a few times without skipping a beat - of course that's no guarantee that it will work perfectly in a future crisis, but it does lower risk while increasing confidence.

Serving up shorten URLs in this manner is not the industry norm, but we believe it gives us the security, should bad things happen, that we can continue to keep our links working. No plan is perfect - it's foreseeable that both our servers and Amazon's servers could go down at the same time – but we believe it's a solid plan commensurate with our budget.

The DiggBar Solution

Digg has received a lot of attention since launching their DiggBar a couple weeks ago. Sites like Engadget block the DiggBar which seems fair - engadget.com is their real estate and blocking it is trivial requiring a simple line of Javascript.

Of course, Digg could retaliate and block any story submitted that links to engadget.com (or, more simply, any link that leads to 207.200.75.1 or 207.200.95.1). But, I could never see this happening nor would it be productive.

However, I can envision a couple solutions that would allow the DiggBar to coexist with sites like Engadget.

1. Simple Solution
One issue that people have with the DiggBar is that users cannot see the domain name of where the content is hosted. To solve this problem, Engadget could set up a DNS alias such that digg.engadget.com points to the Digg URL.

This is a technique we allow people to use at Adjix. For example, the following two links point to the same resource located on ps-enable.com's servers:
http://adjix.com/35zs
http://partner.ps-enable.com/35zs

Note, in the ps-enable.com link, that the URL in the address field of your web browser corresponds the actual domain of the content. Now imagine replacing the ad, at the top of the page, with the DiggBar.

When using this solution, Digg could simply redirect http://digg.com/1234 to http://digg.engadget.com/1234. Although this isn't a perfect solution it does give the reader an idea of which website has published the content.

2. Partnership Solution
A more elegant solution would be if websites partnered with Digg. Digg could provide their partners with a JavaScript snippet to drop on their website much like Google AdSense.

In this solution, a Digg URL, such as http://digg.com/1234 would redirect to the partner's website which would display the DiggBar "organically". In other words, the DiggBar would be served up by the site where the content is hosted.

This partnership solution is win-win since the content provider gets the traffic from Digg and the DiggBar is displayed at the top of the page making it easier for people to digg stories.

Adjix Links on Twitter

If you've been posting Adjix links to Twitter, you'll notice something interesting. Within a minute or two of posting a link ten or twenty bots will click on your link to see where it leads. This is most noticeable with Adjix links that don't contain any ads since they are simple redirects.

Due to Twitter's popularity and API every time a link is posted to Twitter it can register dozens of clicks from these third party bots which automatically index every tweet. These are usually easy to spot, when reviewing your Adjix link stats, either due to their host name containing something obvious like crawl or bot or due to the fact that they usually don't have a referrer.

So, even though it seems that your links, posted to Twitter, receive a dozen clicks off the bat - it's not always people who are clicking on them.

Guy Kawasaki Touts Adjix During Keynote

Guy Kawasaki, who is a big fan of Adjix, touted our virtues when he gave the keynote speech at last week's Search Engine Strategies conference in NYC. Although he mentioned Adjix several times, the best segment is from 22:20 - 24:30.
http://adjix.com/423y

Data Migration

First, let me apologize for today's downtime. I cannot make any excuses and I take full responsibility for today's service interruption. While the Adjix service was down, we executed our fallback plan which enables adjix.com and ad.vu links to continue to redirect users properly. This plan uses Amazon's servers to serve up Adjix links, directly, forgoing ads and link click data collection.

We've collected a lot of data since we launched last August and we've begun implementing a data migration plan. Today was the first, big, step. You should notice that the Adjix website is more responsive, now. However, due to the massive size of the data, we've decided to bring Adjix back on line while we continue to migrate and archive the detailed link click (hit) data. The archiving process will take some time (days or longer) and you'll noticed that your link click details are not available, however, your link click totals should be correct.

I appreciate the messages that have been sent to us and thank you for your continued support.

Sincerely,
Joe Moreno
President
Adjix

Too Much Online Advertising

Sometimes, websites go a little too far with their online ads - to the point that you can't find the content you're looking for; or even sort the content from the ads.

A couple months ago, my wife and I visited my family, back on Long Island, for the Holidays. We had dinner at the Maine Maid Inn, in Jericho, NY, which was built in 1789. This afternoon, my mother and sister contacted me to tell me they had heard that the restaurant had closed down. But, they couldn't find any information about it.

My first stop was the leading Long Island newspaper, Newsday. I ran a search for the Maine Maid Inn and I was surprised that I couldn't find anything about it. So, I turned to the best place to get real time information, online: Twitter.

Here is what I found (click image to enlarge):
Hey, wait, there was an article published online by Newsday. Why didn't I find it? Well, I did, but didn't know it.

Can you find the article here? (click image to enlarge)
I'm sure, now that you know the search results I was looking for are really on that web page, you can see the article at the very bottom of the page. Unfortunately, for me, it wasn't very obvious. Should the Favorite Inns Guide, The Match, and the latest Newsday corrections really be the Top Results? Of course not.

Obviously, well placed ads are the key to getting results - unless you simply intend to bombard your viewers with as many ads as possible. There are more than half a dozen ads on that Newsday web page - I'm sure six, well placed ads, could be more than ten times as effective.

Twitter Search
If you search for most anything on Twitter you'll find what people are saying about it right now - and what they ever said about it. Next time you're watching a significant event, live, on T.V., do some searches on Twitter - you'll be amazed.

Update 3.1.2009
I just read this article about Newsday - they will probably end free access to current news articles on their web site. I just don't think they get it.

Cheers,
Joe

Security Hole Found, Fixed, and Deployed

Just before 10 am PST, today, a security hole was discovered, by chance, in Adjix2Twitter by Sam Nguyen which allowed him to post this tweet to Guy Kawasaki's Twitter timeline. I'd never heard of Sam or his company before today - he is the CTO at InsideWork which "infuses business innovation with biblical insight".

Since Twitter is about as real-time as it gets, the following happened within an hour of the problem being discovered by Sam:

1. I saw the tweet as soon as it was sent and I immediately reviewed the logs to discover that Guy hadn't posted it from his own Adjix account.

2. NEENZ, who is Alltop's Chief Evangelist, DM'd me about the tweet and called Guy.

3. A number of Guy's followers @'d him regarding the tweet - and many also RT'd it, seemingly "in the blind".

4. Twenty minutes after Sam discovered the problem he sent an e-mail to me outlining what he had done and I called him to get the details.

5. Guy, who was in a meeting when this happened, called me after the meeting to find out what was going on and what he needed to do.

6. Guy disavowed the tweet and proclaimed his love for Adjix.

Security Hole Details
While no one likes bugs, the one that Sam found was reproducible which makes it easier to fix.

To reproduce the problem, someone only needed to attempt to "reshrink" an Adjix link using Adjix2Twitter. Adjix2Twitter prevents an Adjix link from being "reshrunk" again and simply returns the original link. The problem was that the Twitter credentials associated with the original link were being used to post to Twitter. No Twitter user info and no Adjix user info was compromised or exposed. Exploiting this security hole only made it possible to post to someone else's Twitter account and it happened exactly once.

To fix this problem, our servers now ensure that the Linker's credentials of the user who clicked on the Adjix2Twitter bookmarklet are used and not the credentials associated with the link.

The Adjix2Twitter fix went live at 12:15 pm, about two hours after Sam first discovered the problem. All is well.

Cheers,
Joe

Mysterious Clicks

Last Friday I was talking on the phone to an old buddy, Andy, from high school. I wanted to show him a web page on the Adjix website so I created an Adjix link (redirect) to it and gave it to him over the phone. He manually typed the link into his web browser without any problems.

After we got off the phone I checked my Adjix link stats and noticed that there were two clicks on the newly created link. One link click was registered about 40 seconds after it was created which was obviously Andy typing it into his web browser. But, about 15 minutes later, while we were still on the phone, there was a second click from this IP address: 150.70.84.43

Who is 150.70.84.43?
This IP address is registered to Japan Network Information Center. This strikes me as a little odd since I live in San Diego and Andy lives in L.A. The only other useful thing I can tell you about this link click is that its user agent was Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1).

How did someone or, more likely, some bot, find out about my link within 20 minutes of it being created? I don't know. My best guess is that maybe Andy has some spyware on his Windows computer.

I'd love to hear you thoughts on this. Tweet them to me @Adjix or via e-mail: MysteryClick@adjix.com

Cheers,
Joe

Twitter Data Analysis

Twitter data analysis of 10.2M tweets from 2.7M users shows Adjix as the 9th most popular URL shortener on Twitter: http://adjix.com/n4e