Crossing the streams – large numbers of Twitter updates

Chris Bilson (@cbilson) had a good description regarding my post about Twitter’s scaling/architecture challenge.

Kevin Rose and Leo Laporte tweet at the same time = crossing the streams”

I dunno if Proton Packs have exponential load challenges, but the end result for a server can feel similar. Is my post I pointed out that Twitter has to determine delivery options and potentially deliver between 100 million and 1 billion updates per day.

But that’s in a day. 1 billion messages in a day are a piece of cake when spread over 24 hours. What if 1 billion messages have to be delivered in an hour? Or all at once?

Take my list of the top-10 Twitter accounts and imagine them all at TED, WWDC, Google I/O, or your local unconference. These ten users, if each sends an update around the same time create 321,928 messages that need delivery (total number of followers for top-10 accounts). This is an awesome amount of message delivery. If those ten users live-blog or get conversational and send ten updates in an hour… 3,219,280 (again, that’s from only 10 users).

I don’t illustrate this to state it’s these power user’s fault. Absolutely the opposite. They’re generating amazing amounts of traffic, which is a wonderful thing, and the algorithms are the problem.

It’s possible to optimize algorithms and modify systems for maximum performance. I bring up Twitter’s challenges because I’m wondering if this is a challenge beyond present day computing.

To open some minds, here’s an impossibility often overlooked: Huge numbers in a deck of cards (just to show impossibilities can stem from small initial numbers).

6 thoughts on “Crossing the streams – large numbers of Twitter updates

  1. You raise a great question with “…if this is a challenge beyond persent day computing”. It might be, although if it is, the catch-up might be fairly quick.

    Good point there.

  2. Right… But Moore's law is X to the power of 2. My calcs point to one-to-many being X to the power of X (would love some feedback here).

    If X is controlled by human population, maybe catching up is possible. But it would also be controlled by Twitter popularity, and it's just now touching the mainstream.

  3. (Shoot, I have to get my slide rule out.)

    Aint Moore's law a bitch? See if you can get Scoble (email) or GG on this one. You might be onto something here.

  4. You raise a great question with “…if this is a challenge beyond persent day computing”. It might be, although if it is, the catch-up might be fairly quick.

    Good point there.

  5. Right… But Moore's law is X to the power of 2. My calcs point to one-to-many being X to the power of X (would love some feedback here).

    If X is controlled by human population, maybe catching up is possible. But it would also be controlled by Twitter popularity, and it's just now touching the mainstream.

  6. (Shoot, I have to get my slide rule out.)

    Aint Moore's law a bitch? See if you can get Scoble (email) or GG on this one. You might be onto something here.

Leave a Comment

Your email address will not be published. Required fields are marked *