Crossing the streams – large numbers of Twitter updates
Chris Bilson (@cbilson) had a good description regarding my post about Twitter’s scaling/architecture challenge.
“Kevin Rose and Leo Laporte tweet at the same time = crossing the streams”
I dunno if Proton Packs have exponential load challenges, but the end result for a server can feel similar. Is my post I pointed out that Twitter has to determine delivery options and potentially deliver between 100 million and 1 billion updates per day.
But that’s in a day. 1 billion messages in a day are a piece of cake when spread over 24 hours. What if 1 billion messages have to be delivered in an hour? Or all at once?
Take my list of the top-10 Twitter accounts and imagine them all at TED, WWDC, Google I/O, or your local unconference. These ten users, if each sends an update around the same time create 321,928 messages that need delivery (total number of followers for top-10 accounts). This is an awesome amount of message delivery. If those ten users live-blog or get conversational and send ten updates in an hour… 3,219,280 (again, that’s from only 10 users).
I don’t illustrate this to state it’s these power user’s fault. Absolutely the opposite. They’re generating amazing amounts of traffic, which is a wonderful thing, and the algorithms are the problem.
It’s possible to optimize algorithms and modify systems for maximum performance. I bring up Twitter’s challenges because I’m wondering if this is a challenge beyond present day computing.
To open some minds, here’s an impossibility often overlooked: Huge numbers in a deck of cards (just to show impossibilities can stem from small initial numbers).
You raise a great question with “…if this is a challenge beyond persent day computing”. It might be, although if it is, the catch-up might be fairly quick.
Good point there.
Right… But Moore's law is X to the power of 2. My calcs point to one-to-many being X to the power of X (would love some feedback here).
If X is controlled by human population, maybe catching up is possible. But it would also be controlled by Twitter popularity, and it's just now touching the mainstream.
(Shoot, I have to get my slide rule out.)
Aint Moore's law a bitch? See if you can get Scoble (email) or GG on this one. You might be onto something here.
You raise a great question with “…if this is a challenge beyond persent day computing”. It might be, although if it is, the catch-up might be fairly quick.
Good point there.
Right… But Moore's law is X to the power of 2. My calcs point to one-to-many being X to the power of X (would love some feedback here).
If X is controlled by human population, maybe catching up is possible. But it would also be controlled by Twitter popularity, and it's just now touching the mainstream.
(Shoot, I have to get my slide rule out.)
Aint Moore's law a bitch? See if you can get Scoble (email) or GG on this one. You might be onto something here.