Crossing the streams – large numbers of Twitter updates

Posted on by Sol Young with 6 Comments

Chris Bilson (@cbilson) had a good description regarding my post about Twitter’s scaling/architecture challenge.

“Kevin Rose and Leo Laporte tweet at the same time = crossing the streams”

I dunno if Proton Packs have exponential load challenges, but the end result for a server can feel similar. Is my post I pointed out that Twitter has to determine delivery options and potentially deliver between 100 million and 1 billion updates per day.

But that’s in a day. 1 billion messages in a day are a piece of cake when spread over 24 hours. What if 1 billion messages have to be delivered in an hour? Or all at once?

Take my list of the top-10 Twitter accounts and imagine them all at TED, WWDC, Google I/O, or your local unconference. These ten users, if each sends an update around the same time create 321,928 messages that need delivery (total number of followers for top-10 accounts). This is an awesome amount of message delivery. If those ten users live-blog or get conversational and send ten updates in an hour… 3,219,280 (again, that’s from only 10 users).

I don’t illustrate this to state it’s these power user’s fault. Absolutely the opposite. They’re generating amazing amounts of traffic, which is a wonderful thing, and the algorithms are the problem.

It’s possible to optimize algorithms and modify systems for maximum performance. I bring up Twitter’s challenges because I’m wondering if this is a challenge beyond present day computing.

To open some minds, here’s an impossibility often overlooked: Huge numbers in a deck of cards (just to show impossibilities can stem from small initial numbers).

Posted in Architecture, Blogging, Database, Micro-blogging, Microblogging, Social Networking, Software Development, Twitter, Web Services

Roz Savage is rowing across the Pacific

Posted on by Sol Young with 2 Comments

Thanks to new Twitter user @waileacapital for posting about a British woman, Roz Savage attempting to row across the Pacific (she rowed across the Atlantic already). If she succeeds she will be the first woman to complete the journey.

Roz blogs on a daily basis during a trip at http://rozsavage.com/blog. Her main site is at http://rozsavage.com. The video above is of her Atlantic trip.

The latest is that she is having trouble with her desalinization system and having trouble getting fresh water.

Posted in Blogging, Energy, Twitter, YouTube

Twitter’s one-to-many scaling impossible?

Posted on by Sol Young with 5 Comments

Twitter has been having all kinds of scaling challenges. There have been hundreds, if not thousands, of posts on the subject. Dave Winer pushed an idea for a decentralized Twitter (and has since admitted the power of Twitter is in its centrality). There is a single, simple, reason for Twitter’s challenges – Math is against them.

The facility of communication on the Twitter service is absolutely outstanding. I’ve written extensively about using it to receive an amazing amount of quality information in my series on flow.

I originally questioned the scaling ability of the service prior to SXSW, but when the service held up I went back to the drawing board to make sure my numbers were correct.

Before continuing, let’s establish the basics about the service so the math will make sense…

  • Each Twitter account can follow any other Twitter account (bear with me and forget those accounts with private updates).
  • Messages travel in one direction, from the updater to the follower.
  • Each account has updates from other accounts it follows placed in its timeline.
  • A Twitter account can selectively receive pushed updates immediately via instant messenger and SMS in addition to having an update added to its timeline.
  • An update added to an account’s timeline may or may not be push based (lets assume it’s demand driven, or pull based).
  • An update sent to an account from an account denoted as SMS or IM announcement is push based (there is no other way to send an update – it must be actively pushed from the server).
  • The mere possibility of an update needing to be pushed requires the system to check with each follower’s settings, thus requiring analysis of each follower for each update.

A warm-up equation

If there are one hundred (100) users and each user follows ten (10) fellow users, and each user sends ten (10) updates per day, assuming all updates are push-based, how many updates are sent?

Continue Reading

Posted in Architecture, Blogging, Flow, Math, Micro-blogging, Microblogging, Social Networking, Software Development, Twitter, XMPP

Scoble and Gary in DC

Posted on by Sol Young with Leave a comment

MCCXXIII

.

Last night I headed down to MCCXXIII, a club at 1223 NW. Connecticut, for a DC Social Media meetup. The meetup featured Robert Scoble (@scobleizer) and Gary Vaynerchuk (@garyvee) as guests.

I’ve been following Scoble more than usual this week, as he’s come to DC to interview politicians and government officials with Rocky (@RocmanUSA) and his son Patrick (@pscoble). It’s been a merging of the blogging tech world and the US government (finally!).

The club was jammed, no room to move, great vibe and so packed the A/C was ineffective. It was awesome to see the turnout and feel the energy towards geeks making inroads to Washington.

Scoble described the experience in a single word, “Incredible.” We talked for a while on East Coast vs. West Coast interviews and the whole crew agreed that it’s been a refreshing experience. There’s nothing wrong with it, but the Silicon Valley CEOs are embellishing the value of their company. Here, it was described, instead of an executive power (pun intended), the power is more pure and the answers more honest.

When Robert described the difference in honesty I did a double-take. Seriously? Whoa. He described a few of the interviews, talking about some of his favorite moments and personal opinions. I’ll let you catch those in the coming days at Scobleizer.com and in video on FastCompany.tv.

A highlight he kept coming back to: His son nailed down the FCC chairman on the topic of porn (it wasn’t on video – wish it was). Robert’s eyes beamed with pride as he described his son having the guts to ask the questions.

Overall, a good meetup and cool to have Scoble out here for a few days, doing something different and interesting.

Existing posts on Scoble’s visit to Washington:

UPDATE: Here’s the link for Gary’s book tour ’08: http://tv.winelibrary.com/gary-vs-book-tour-2008/

Gary Vaynerchuk is continuing his book tour, visiting the Rockville, MD, Barnes & Noble for a signing this morning at 9:00AM.

Posted in Blogging, Economics, Education, Energy, Social Networking, Twitter, Utterz

Mobile phone GPS a security risk? Only to those who follow you

Posted on by Sol Young with Leave a comment

Steven Hodson, in a post over on Mashable, describes security risks and the waste of information that mobile phone GPS use brings (when pinpointing and announcing our locations). He poses some extremely valid points in regards to announcing one’s geolocation via Twitter, Brightkite, or FriendFeed being useless noise.

Telling people via a highly conversational medium such as instant messaging or an SMS text that you are currently at 13th St and Ash Lane is nothing more than noise. It’s a waste of precious conversation. To a few of your closer friends it may be relevant, such as a buddy that would meet you for coffee. But for the masses, it is unimportant and you’re guilty for wasting their time.

I’m guilty of this lately. I’ve been trying various mobile phone GPS services. It’s been fun and interesting, but I’m in agreement with Steven about this announcement being a waste (at least if it’s without background information). Steven doesn’t mention it, but my thought on optimal geolocation announcement is in a widget placed on one’s blog. It’s there for interested followers, but not intrusive or annoying.

But that’s where we agree. He describes broadcasting one’s geolocation as a security risk and I strongly disagree. Yes, there are some situations where it is. US soldiers in Iraq will not benefit from this feature. Folks in witness protection programs, runaways, victims of domestic violence, those being stalked, cheating spouses, and those in organized crime probably won’t either.

The typical citizen without conflict is not at risk. It’s easy to figure out when someone is normally at work, so knowing an optimal time to break in to someone’s home is already simple. It’s easy to find a person in a public place, so it’s already easy to find the optimal time to commit physical harm.

Note: If one is being stalked or believes him or herself to be in a situation where announcing location is dangerous, it’s simple to turn the feature off.

As I’ve said before, people are inherently good. They don’t go around looking for someone to damage or rob. There are some people who commit these crimes. These people use crow-bars instead of Facebook, and are stopped by alarm systems and deadbolts rather than a lack of geolocation data.

Posted in Etiquette, Facebook, Flying, FriendFeed, Geolocation, GPS, Jaiku, Life Streaming, Micro-blogging, Microblogging, Pownce, Security, Social Networking, Twitter, Wearing