In 2010, we need to find a better way to filter the web. It’s growing exponentially every day to the point that only the largest server farms can keep up.
Twitter’s API can’t keep up with it’s own traffic. Soon this will change when the firehouse is opened up to everyone but it will just push the problem further downstream. Developers are eager to have access to the firehouse of data but they won’t be able to process it all, nor should they try. And this is only for one piece of the real-time web puzzle. Factor in Facebook, Google Wave, Linked-In’s upcoming API, many more, and it becomes next to impossible for one company to filter and analyze everything.
To resolve this problem, we need micro filters.
What is a Micro Filter?
A micro filter is a filter that has a unique purpose and is reusable and available to anyone.
One example of a micro filter is a Twitter list. These lists are filters that web applications can use to narrow the firehouse and make information gathering manageable. But there’s one problem. Twitter lists don’t filter the information in a meaningful way. You can’t grab every Twitter list on marketing and gather all the marketing tweets. A marketing twitter list can be as diverse as Twitter itself and can overlap with many other lists outside of marketing.
This is why we need multiple micro filters to get the information we want. A series of filters – when put together – would narrow the focus of information to the data you need for your web application or research project. Running your marketing twitter lists through a marketing filter would narrow the focus and give you the marketing information you need.
Creating Micro Filters
Creating micro filters is very complex. I used Twitter lists as an example but this is one of the easier filters to build. The complexity increases when you try to create the “marketing” filter in the example above. How do you know what information in a Tweet is related to marketing?
There are several ways to do this:
- Hash Tags: Hash tags are great identifiers but they’re not popular enough to filter on. Too much information would be lost.
- Open API’s: Take the links from each Tweet, convert the URL to it’s long format, reference it in Delicious, and look for marketing tags. This works but it has a couple of downsides. First, it requires a lot of processing time. Second, the link may not be tagged in Delicious yet.
There isn’t a perfect solution but it’s clear that a combination of tactics are needed to build this “marketing” filter – tactics that go well beyond individuals categorizing other individuals in a social networking platform such as Twitter lists.
Further, several micro filters could be put together to keep narrowing the focus. You could add a third filter to the example above that shows all marketing information shared within 5 miles of you. This location filter would be the third micro filter and it could be used an many different situations.
In 2010, I expect to see more filters become available to help people focus on the topics that interest them most. Looking at Twitter, it’s clear that filtering is going to become the next big development as people gather more followers, share more information, and expand their presence across more social media platforms.
Currently, Twitter is an unreliable platform for contacting people as the API can’t handle the streams of information going to its most popular residents. Further, at close to 300 million Tweets per week, there’s a lot of great information getting lost in the noise and this isn’t just an issue on Twitter. It’s happening everywhere which is why micro filters are the future of the web.