There is a metric used for scholarly journals called the impact factor (IF). In the world of medical and professional journals, the criteria for influence is citations. Similar to how much of the social media world uses inbound links – the citation metric carries significant weight in scientific and academic communities. It also raises significant debate from critics who feel it is an incomplete measure. A few shortcomings of the impact factor include:
- Ability for original authors of an article to affect the IF by citing their own publication in other articles
- Too short a time window for citations, ignoring ongoing citations to older "classic" articles
- Only counting the frequency of citations and disregarding the prestige of the citing journals
- IF is only relevant for certain fields like science or economics, but not for other fields like literature or less citation based
BUT it is the best that the scientific community has and it is used extensively. Does any of this sound familiar? Many of these are the same issues that bloggers are raising with services like Technorati and sparking discussion about what influence and authority really means in social media. I like the word "impact" because it seems concrete. Impact seems measurable. But what is the model that should emerge for how to measure it in relation to words already in use like authority or influence? I think part of the difficulty in finding the right model comes from not really understanding the right elements to measure. Right now, the most popular metric for bloggers is Authority from Technorati, and this is based mostly on volume. Most people agree we need more sophistication, but rather than massaging existing models … we need to have a common understanding of what the most important criteria are to measure to really understand the impact factor of a particular blog.
To that end, here are the five areas of measure I think we should be focusing on:
- Volume – how many unique mentions did it get?
- Prestige – how influential were each of the mentions?
- Depth – how detailed were each of the mentions?
- Time – over what period of time did the mentions happen?
- Reputation – how well regarded or well known is the author in their area?
What do you think about these? If there were a measurement model that could account for each of these criteria, would it paint a complete picture of influence and allow comparison between blogs? More importantly, has anyone already sorted this out and managed to built a model to go beyond volume and encompass some of the other areas?
Weird. I just wrote a post last night on words like “Authority” and “Impact”. We absolutely need a better model than Technorati. Amongst your model, I am not sure “prestige” belongs. It is not a good proxy or synonym for influence. Ultimately influence must be measured at the impact point – did it change/inform attitude or behavior. That is asking a lot from a methodology based upon observing blogs.
Those metrics aren’t elusive just for blogs. The search industry has been trying to nail down an algorithm that takes into account all those criteria (it’s plural, fyi) for general search results too.
But how do you quantify factors as subjective as prestige and reputation? A good example is Scoble. Gets loads of traffic, comments and citations around the Web, but they swing from one extreme in to the other in terms of quality, legitimacy and relevance. You could probably find as many who loathe Scoble as admire him. I agree with John that “influence” gets a little closer to the mark, esp. if you simply want to determine how much buzz a particular blogger is getting, for better or worse.
We need something. None of the current mechanisms seem legitimate, and the Ad AGe 150 really seems to be antiquated with both Bloglines subs and Todd’s content benchmark.
I like the Friendly Ghost’s listing only because it weighs 7-8 factors. This creates a higher likelihood that if one factor is flawed it can be counter balanced by others.
Technorati’s search function produces below average results and I don’t have too many colleagues that pay much attention to it’s rankings.
I am in a similar boat with John. I’m not too sure about the prestige criteria. In an effort to keep everything MECE, I would eliminate prestige as it seems to overlap with reputation.
Thanks for putting in the time to organize this. What are the next steps- applying a metric to each criteria?
When you chat with your friends and colleagues, do you measure the impact factor of what your saying or what others might say about your conversation?
The whole blogging and social media movement started as a way to engage conversations, loosen up things between people and brands and bring them togheter.
Now we want to measure that?
===================
part 2
ok, I know – one client just asked me how should we measure the impact of a UGC website we developed for them.
But just when the Internet was becoming nice and beautiful, the clients and money came in…and compromises…
It’s like asking “how much do you love me customer?” / “oh, brand, i love you so much I would by your products every day…”
Rohit, I love this concept of Impact Factor. We’re starting up the Social Media Metrics Blog again after a long hiatus and your post factors in — https://goldgroup.blog.com. The question is can we ‘make up’ a metric that captures a sense of impact using only the tools available today? Remember that Page Views, Unique Visitors and the like were once made-up concepts too.
It is quite interesting but Technorati is too late. I agree with Nick regarding the blogs. It is true that Impact Factor effects a lot on Social Media. We need to have an objective measure, and have a wider acceptance than any of the alternatives of social Media.