Tuesday, July 7

Measuring Community Health

While we have been running metrics in our communities for 'measuring community success', we (like everyone else) have struggled with defining any form of notion of how we're doing as a company against other peers or even other companies in other areas.

The main problem is what metrics do you compare on. We can't compare with someone the size of say, Dell using registrations, as they have a user base that is in another universe compared to us. Likewise comparing number of posts in say a movie forum is pointless as the level of passion is completely different.

This wasn't just our problem, it has existed across the industry. While traditional web measures are useful (users, registrations, posts, threads) these have exactly the same major flaw as traditional web stats. There is no measure of success/interaction in quantitive figures.

Lithium has been working with some mathematicians, particularly Dr Michael Wu (twitter: @mich8elwu), to help introduce a measurement scale that works for each community, regardless of size, and allows you to not only measure your community health over time, but also compare its success to any other community.

What is CHI?

Called 'Community Health Index' or CHI for short, it is a 1000pt scale that takes input in six key areas:
  • Members
  • Content
  • Traffic
  • Responsiveness
  • Interaction
  • Liveliness
The first three measure traditional 'web' quantitive areas - reg users, posts, replies, visits etc - and then factors them against the 'Community' measures of time to respond, distinct users vs # of conversations, and actual posts vs expected posts, etc.

On their scale of 1-1000, below 400 is cause for concern, 400-650 is average, and 650+ is very good. My community currently has a rating of 567, which is doing well, but has room to do more.

Our primary concern area is Liveliness - the amount happening per blog or forum.

What this means

We now have a concise way of measuring our community health, expressed in a single number. When we delve into the 6 areas of measurement it provides us with very specific targets for projects to improve our health.

We will have a CHI report on a regular basis from our Lithium account manager, and that keeps us abreast of whats going on. But it gives us more than that. I now have the ability to target particular metrics in my promotion or tuning of a community. And for a Community Manager, that is gold.

Coincidentally Dr Wu is running a webcast in August to talk about this.

1 comment:

Anton Chiang said...

Great write up Bladefrog.. CHI is certainly going to help community managers scientifically measure community health. This used to be something that's "a good community manager's gut instinct", but I'd much rather have math and science back up my instinct :)

It's great to see Lithium making CHI an open standard, I look forward to seeing this measurement system being adapted by more community platforms out there