The 2,000 follower limit, it would seem, was put in place to prevent mass following and spam on Twitter. This was pretty frustrating for me since I fell in to their beyond-the-limit zone (I followed over 6,000 people because I loved the information, but couldn’t add any more).

I’m not complaining too much, as I’m enjoying the more traditional use of my Twitter account for now, but this is a ridiculously short-sighted fix.

I haven’t seen much attention drawn to the following facts (pun wasn’t intended):

1. People are more likely to recipricate a follow request from someone with a low following/friend count.
2. There isn’t a legitimate way to prevent someone from having multiple Twitter accounts (accounts are tied to email addresses).
3. The Twitter API limits are based on account, not where the call is coming from (one server can make many requests on behalf of other accounts).

From the above simple observations, one can see the easy way to follow an unlimited number of people.

1. Create a large number of accounts.
2. Follow a smaller number of people with each account (you’ll have better reciprocation).
3. Follow a lot of people (the API limitations will apply per account, so your follows-per-hour will actually be quite large).

The people running Twitter are great. They’re really trying to do the right thing. So maybe I’m completely wrong when I anticipate the above and say that this looks like a Facebook move. Facebook’s 5,000 friend limit works for Facebook. Facebook’s API is advanced and robust and complicated enough to not get terribly nailed by multi-account mass spam following.

Additionally, the information load on Facebook is different. You get a clear picture of who a person is that is friending you. You’re given enough information to make a decision. On Twitter, this isn’t the case.

So what’s going to happen?

1. Spammers are already adapting to the limitation, as described above.
2. Tweeple will stop trusting low follow-count users (do you trust an eBay user without feedback?)
3. Twitter’s servers will still be inundated and over capacity.

I blame it on Scoble…

“What have I done!?”

I’ve gone back to the traditional use of Twitter. The method more than 95% of the userbase uses it for. I now use it to stay in touch with the people I’ve met and know personally, rather than using Twitter as a medium for info aggregation. It’s not possible to use Twitter how I did in the past.

If you know my series on flow (it kicked off here), you know what I was doing and how cool it was. I got the idea partially from Robert Scoble’s entry, The Secret to Twitter. His use was brilliant and it worked amazingly well!

Back in March of ‘08 I began following any interesting person I thought to be intelligent and putting out informative tweets. Primarily I found people in the software development, new media, aviation, library science, and management arenas. I ended up following 6,218 people at the high (last week). Everyone’s updates were viewed in IM and I would see an amazing flow of information.

Usually hundreds of tweets per minute, forcing me to read very quickly and get a quick read on the blogging, technology, and media areas in a short period of time. It allowed me to find articles and posts that would have filtered in slowly on RSS (arguably, if I had more than my 632 RSS feeds I’d find more information here, too).

It was great. Flip on iChat over breakfast and watch the flow while eating granola and yogurt. An ideal start to the day.

But in the last week I’ve culled over 4,000. The removed are people who don’t follow me and who I never met in real life. The chance of our interaction is very small, and if we meet I’ll follow.

I’m looking forward to having more intimate interaction with friends and followers. Focus will shift more towards FriendFeed and Google Reader (RSS).

Twicecream – a fake service to demonstrate a point about single sign-on…

In web 2.0 there is a determination to screw up potentially great services. It’s my number #1 pet peeve with software development these days. Here’s a fictitious example of a service you might create…

You’ve built a service that automatically Twitters your geo-position and the name of an ice cream parlor when you’re in front of it. Your phone buzzes when an ice cream parlor is detected and begins sending photos to SnapTweet and TwitPic, including Zagats ratings and commentary. Other patrons respond back and generate conversations. This is your social network: Twicecream – a social network for twittering ice cream enthusiasts.

In front of Ben & Jerry’s on the Wharf, Zagats 4-stars, pics: http://twicecream.com/abc123

Congratulations! You just failed.

You didn’t fail by creating a service few would use. You failed because you didn’t utilize the authentication mechanism your patrons preferred. You built an unnecessary barrier to your garden by requiring an unnecessary account creation. Don’t do this, it’s arrogant and inefficient.

Your patrons have Twitter accounts. Twitter has an API. Your service should have asked the patron to log in with their Twitter credentials.

This isn’t just for social networking. This goes for all web services. SaaS solutions that require secondary account creations are a bad idea. Single sign-on, whenever possible, should be used.

The whole idea is to simplify access to what the customer needs. If you’re requiring unnecessary account creations, you’re screwing it all up.

Had an awesome day – time to share!

• 12seconds.tv/channels/solyoung
• First 12second introduction
• TechCrunch link
• WCU post

On Twitter, the numbers for friends (people you follow) and followers (people who follow you) are being misreported. The most common tweet today is about one’s follower count dropping off. This is telling of your personality, and not in a good way.

If you’re complaining about your follower count dropping off without your realizing your friend count dropped too, you’ve probably only been paying attention to building numbers. You’re also describing to the world that you didn’t care about losing touch with your friends.

For the record, I first noticed my friend count dropped. Over the year or two on the service, I’ve built up a friend list of ~6200 extremely interesting people (~2100 follow me). When I saw my friend count dropped, I checked my followers to verify. My tweet was, “Wow – number of people I follow and who are following me just dropped by more than 1000 each. Not good!”

While they reboot the service and get your numbers back to normal, take a moment to consider what matters. If you’re that important, they’ll find you again.

Ross Miller at FriendFeed responded to my question about FriendFeed del.icio.us feeds not updating. It seems del.icio.us is blocking mass spidering of their site.

Ross’s email:

“Hi Sol, del.icio.us places restraints on our ability to crawl their site. So if for some reason the item is missed, it becomes very hard for us to retrieve the del.icio.us link. Unfortunately for now the refresh button is your best option when this happens. Sorry for the inconvenience and we hope to get this resolved with del.icio.us soon.”

Looks like this is a restriction on del.icio.us’s end.

Chris Bilson (@cbilson) had a good description regarding my post about Twitter’s scaling/architecture challenge.

“Kevin Rose and Leo Laporte tweet at the same time = crossing the streams”

I dunno if Proton Packs have exponential load challenges, but the end result for a server can feel similar. Is my post I pointed out that Twitter has to determine delivery options and potentially deliver between 100 million and 1 billion updates per day.

But that’s in a day. 1 billion messages in a day are a piece of cake when spread over 24 hours. What if 1 billion messages have to be delivered in an hour? Or all at once?

Take my list of the top-10 Twitter accounts and imagine them all at TED, WWDC, Google I/O, or your local unconference. These ten users, if each sends an update around the same time create 321,928 messages that need delivery (total number of followers for top-10 accounts). This is an awesome amount of message delivery. If those ten users live-blog or get conversational and send ten updates in an hour… 3,219,280 (again, that’s from only 10 users).

I don’t illustrate this to state it’s these power user’s fault. Absolutely the opposite. They’re generating amazing amounts of traffic, which is a wonderful thing, and the algorithms are the problem.

It’s possible to optimize algorithms and modify systems for maximum performance. I bring up Twitter’s challenges because I’m wondering if this is a challenge beyond present day computing.

To open some minds, here’s an impossibility often overlooked: Huge numbers in a deck of cards (just to show impossibilities can stem from small initial numbers).

Twitter has been having all kinds of scaling challenges. There have been hundreds, if not thousands, of posts on the subject. Dave Winer pushed an idea for a decentralized Twitter (and has since admitted the power of Twitter is in its centrality). There is a single, simple, reason for Twitter’s challenges – Math is against them.

The facility of communication on the Twitter service is absolutely outstanding. I’ve written extensively about using it to receive an amazing amount of quality information in my series on flow.

I originally questioned the scaling ability of the service prior to SXSW, but when the service held up I went back to the drawing board to make sure my numbers were correct.

Before continuing, let’s establish the basics about the service so the math will make sense…

• Each Twitter account can follow any other Twitter account (bear with me and forget those accounts with private updates).
• Messages travel in one direction, from the updater to the follower.
• Each account has updates from other accounts it follows placed in its timeline.
• A Twitter account can selectively receive pushed updates immediately via instant messenger and SMS in addition to having an update added to its timeline.
• An update added to an account’s timeline may or may not be push based (lets assume it’s demand driven, or pull based).
• An update sent to an account from an account denoted as SMS or IM announcement is push based (there is no other way to send an update – it must be actively pushed from the server).
• The mere possibility of an update needing to be pushed requires the system to check with each follower’s settings, thus requiring analysis of each follower for each update.

A warm-up equation

If there are one hundred (100) users and each user follows ten (10) fellow users, and each user sends ten (10) updates per day, assuming all updates are push-based, how many updates are sent?

The answer is 10,000 – each sent update (100 users x 10 updates) is forked out to 10 followers who have requested push updates. This is a very large number of updates to send out via SMS or IM compared to the base of users.

A very important fact: It doesn’t matter if a user follows with the intention of receiving an SMS or IM update. The possibility of an updating needing to be pushed requires Twitter to examine every follower when an update is received.

From 1999 to 2004 I worked as a software engineer at Mplayer (who then changed names to HearMe, who then sold their video technology to LIvVE, who then was bought by GameSpy).

As with any chat room, the scaling is similar to Twitter. We had to restrict rooms to 500 users (and had insanely reduced reliability as we approached 500). As shown with the warm-up, each message is forked out to every user. We capped at 500 in a chat room because each user in a room contributes some amount of messages, and therefore as users join a room the traffic grows exponentially.

Official and unofficial numbers

According to Twitter’s blog post with stats, 50% of the Twitter population has 10 followers. 10% of users have 80 or more followers. According to TechCrunch’s research there were 200,000 active users posting 3,000,000 updates per day (as of the end of April 2008). The average Twitter user posts 15 updates per day (3,000,000 divided by 200,000 = 15).

We’ll use Twitter’s percentages in their blog post and combine them with TechCrunch’s numbers. From this, we know there are 100,000 daily users with 10 followers and there are 20,000 people with 80 or more followers. To keep things simple, we’ll leave the other 80,000 daily users out of the equation for now.

• 100,000 users x 15 updates per day x 10 followers = 15,000,000
• 20,000 users x 15 updates per day x 80 followers = 24,000,000

For laughs, let’s put in the top 10 Twitter accounts with the most followers (beware, there were fights over this).

1. Kevin Rose x 15 updates per day x 46,646 followers = 699,690
2. Leo Laporte x 15 updates per day x 44,948 followers = 674,220
3. Barack Obama x 15 updates per day x 42,201 followers = 633,015
4. Alex Albrecht x 15 updates per day x 30,348 followers = 455,220
5. Jason Calacanis x 15 updates per day x 28,773 followers = 431,595
6. Robert Scoble x 15 updates per day x 28,037 followers = 420,555
7. Mars Phoenix (rover) x 15 updates per day x 26,828 followers = 402,420
8. Veronica x 15 updates per day x 26,199 followers = 392,985
9. John C. Dvorak x 15 updates per day x 24,102 followers = 361,530
10. MacRumors x 15 updates per day x 23,846 followers = 357,690

Total of average users + top 10: 43,828,920 updates delivered per day.

And that’s half of the Twitter user base, mixed with a tiny fraction of the users who have large number of followers. Realistically, my estimate above is less than 10% of actual traffic because I’ve left out the 40% and have not included the thousands of highly popular users with more than 80 followers. Additionally, the number of followers for the people in the top-10 has grown between 50 and 100 percent since the end of April! (Twitterholic)

This puts Twitter’s actual message analysis and possible delivery between 100,000,000 and 1,000,000,000 per day.

This also does not include a single page view or web service call to their servers. Those alone account for a huge amount of Twitter’s traffic.

Compared to IM traffic

Back in 2005 (ZDNet) there were 13.9 billion instant messages sent per day, with estimates of quadrupled traffic by 2009 (46.5 billion). Instant messaging is divided up among a few primary services and IMs are one-to-one. According to Wikipedia, AOL AIM has 53 million users. If Twitter became as widely used as AIM, it would grow 265 times (53,000,000 divided by 200,000).

Take our findings for the number of delivered (or analyzed) updates on Twitter and multiply by this growth and you find Twitter has to be capable of delivering between 26.5 billion and 265 billion updates (probably much closer to the latter).

How can Twitter scale?

Decentralized XMPP is probably the answer, but I don’t really know. I can see the problem though. If they grow to having one million daily users, they have between 500 million and 5 billion messages to deliver. If they grow as popular and as relied upon as AIM, they’re staring straight in to their own exponential order of magnitude.

Steven Hodson, in a post over on Mashable, describes security risks and the waste of information that mobile phone GPS use brings (when pinpointing and announcing our locations). He poses some extremely valid points in regards to announcing one’s geolocation via Twitter, Brightkite, or FriendFeed being useless noise.

Telling people via a highly conversational medium such as instant messaging or an SMS text that you are currently at 13th St and Ash Lane is nothing more than noise. It’s a waste of precious conversation. To a few of your closer friends it may be relevant, such as a buddy that would meet you for coffee. But for the masses, it is unimportant and you’re guilty for wasting their time.

I’m guilty of this lately. I’ve been trying various mobile phone GPS services. It’s been fun and interesting, but I’m in agreement with Steven about this announcement being a waste (at least if it’s without background information). Steven doesn’t mention it, but my thought on optimal geolocation announcement is in a widget placed on one’s blog. It’s there for interested followers, but not intrusive or annoying.

But that’s where we agree. He describes broadcasting one’s geolocation as a security risk and I strongly disagree. Yes, there are some situations where it is. US soldiers in Iraq will not benefit from this feature. Folks in witness protection programs, runaways, victims of domestic violence, those being stalked, cheating spouses, and those in organized crime probably won’t either.

The typical citizen without conflict is not at risk. It’s easy to figure out when someone is normally at work, so knowing an optimal time to break in to someone’s home is already simple. It’s easy to find a person in a public place, so it’s already easy to find the optimal time to commit physical harm.

Note: If one is being stalked or believes him or herself to be in a situation where announcing location is dangerous, it’s simple to turn the feature off.

As I’ve said before, people are inherently good. They don’t go around looking for someone to damage or rob. There are some people who commit these crimes. These people use crow-bars instead of Facebook, and are stopped by alarm systems and deadbolts rather than a lack of geolocation data.

Most smartphones slated for release over the next 12-months include a GPS receiver, built in. After that, it will be a marked failure to not include a GPS in a phone. The functionality that comes with GPS is outstanding – mapping, directions, location based experiences, etc. We’re about to enter an age of advancement in technological capabilities that we’re just beginning to imagine.

Consumers are moving to smartphones. The hottest smartphones (iPhone 3G, BlackBerry Bold 9000, most of Nokia’s Symbian and HTC’s Windows Mobile offerings) all include GPS and an exposed API for developing applications utilizing their hardware. Anything people can conceive of for location based mashups will be coming (more on these mashups in later posts)…

Here’s a first application…

BlackBerry is a leader in mobile phone GPS. Recently a few services that announce the location of one’s phone emerged. Initially these were billed as a sort of low-jack for one’s phone, a security service for the insecure (or those who want to spy on their kids, etc).

I decided to try a few of these. Most felt slimy, like, “you always know where your phone is, and you could also know where your wife is!” … I don’t know about you, but my phone is loyal and doesn’t run off with strangers… And I trust my wife far more than a phone.

My goal with trying these services was to mash Twitter, Pownce, Facebook, and other social networking services with my location. Such a mashup will allow me to share my real-time location with all friends. I came across BlackberryTracker.com. Much like the others, the idea is to provide you with the location of your phone. However, they have something the other’s don’t… Facebook and Google Earth integration (as well as a drop-dead-simple semi-RESTful API).

Friends can pinpoint me down to the meter on my Facebook profile, updated every 30 seconds. To be honest, it feels strange to openly publish this data. Security, and lack thereof, has us believing we shouldn’t share such information. But this fear is caused by the exception and not the rule. And in reality, my location in public isn’t private. Additionally, there are laws and common courtesies we live by, and I trust that people are inherently good.

Soon these services will be in the mainstream. Everyone will be able to pinpoint the location of anyone. Let me emphasize that… Soon everyone will be able to pinpoint the location of anyone. Not publishing your location will be like not having a mobile phone.

Here’s a breakdown of the sites and social networks I frequent. Numbers indicate my personal value (10 being daily use and highly valuable and 1 being a land-grabbed account and following the service’s popularity). Plus (+) indicates rising popularity and minus (-) means I’m losing interest:

• ActiveRain – sol – 3+
• Bebo – solyoung – 2-
  
• del.icio.us – solyoung – 9+
• Digg – pytchfork – 4-
  
• DOPPLR – solyoung – 3+
  
• eBay – pytchfork – 2-
  
• Facebook – 604231141 – 8
• Flickr – solyoung – 8
  
• FriendFeed – sol -8+
• Friendster – solyoung – 1-
  
• Google Calendar – solyoung – 10
• Google Reader – solyoung – 10-
  
• Jumpcut – solyoung – 2+
  
• Last.fm – solyoung – 7+
• LinkedIn – solyoung – 7-
  
• LiveJournal – solyoung – 1-
  
• Multiply – solyoung – 2+
  
• MyBlogLog – solyoung – 4+
  
• MySpace – pytchfork – 5-
  
• Plaxo – Sol Young – 4+
  
• Pownce – solyoung – 3-
• Seesmic – sol – 4+
  
• StumbleUpon – solyoung – 5-
• 30 Boxes – SolYoung – 2-
  
• Tumblr – solyoung – 2-
  
• Twitter – sol – 9
• Upcoming – 221289 – 7+
  
• YouTube – pytchfork – 6
• Zillow – solyoung – 6
  

I’ve been using FriendFeed and del.icio.us more and more lately. Viigo (one of the best mobile RSS experiences available) has direct push to del.icio.us. It’s far superior to the iPhone’s mobile Google Reader sharing experience, so I’ve been happily consuming and sharing.

But FriendFeed isn’t updating these del.icio.us posts. It would be understandable if everything weren’t updating, but it seems to be del.icio.us entries. Flickr, blog entries, Google Reader, and Twitter items are nearly instant.

To get around this one can view his or her page, edit/add services, click the del.icio.us item, and finally click ‘refresh’ – but that’s hardly worth the effort.

Is FriendFeed feeling growing pains? Bret Taylor mentioned in a comment to Robert Scoble (who had similar updating problems) FriendFeed had a hiccup during a spider.

Personally, I doubt this was anything more than a technical glitch, but geez… The purpose of FriendFeed is to share what we’re doing on the Internet. If things aren’t pulling through, it’s a critical issue.

• For fellow FriendFeeders: http://friendfeed.com/sol
• For friends on Del.icio.us: http://del.icio.us/solyoung

Quick Links to the streams:

• Flickr Photostream
• Twitter
• Utterz

My plan was to Twitter my progress and TwitPic/Flickr the pics out to my followers. But I woke up at 3am from a caffeine rush and a thought of typing for an hour becoming a nightmare – and boring. Utterz.com, a service doing pretty slick mashups of audio/video/text/photographs/etc, while harnessing APIs from pretty much every popular social networking service, caught my eye (more on Utterz later).

Here’s the day – check the Flickr photostream and the Utterz links below for my audio commentary while I running…

1. Lined up and ready to go
2. Started!
3. Mile 1
4. Mile 2
5. Mile 3
6. Live music between mile 3 and 4
7. Mile 4
8. Mile 5
9. Passing Ed Rendell, governor of PA
10. Mile 7
11. Mile 9
12. Finished!

About Utterz: You start an utter by calling or emailing. Either one initiates it. After that, you have 10 minutes to add more content to the utter. Utterz automatically adds audio from a call or content from an emailed pic/video/audio/text to create an utter similar to what I created above. The call-in feature (dial, press 2, talk, hang up) is extremely simple.

Now, none of the above would be impressive except that Utterz.com is doing it right. This is how services today are supposed to work. Utterz is tied in to all the social networking and blogging services. Any new utter is announced on Twitter, Pownce, Jaiku, or even your WordPress/Blogger/MoveableType/etc blog (hosted or self-hosted – COOL!)