Moderator performance reports, and niche owner role in moderator management

Service: Narrative

Moderator performance reports, and niche owner role in moderator management

Service: Narrative

Hey!

In conversation with @Banter and @Emily Barnett and others, I came up with some thoughts that could improve moderator performance, and also give niche owners a means to address moderator underachievement effectively, within parameters controlled by the system (to prevent abuses by owners).

The thoughts revolve around moderator performance reports periodically sent to owners, and depending on the system-determined analysis for each moderator, opening up options for the owner to address issues.  They also include the concept of owners being able to have some degree of control over whether moderator rewards are somewhat performance-based, or if they all get paid equally based on the system's algorithm.  I've always assumed that algorithm would at least take into account the volume of work done by a moderator so that those mods who work through 100 posts per week would get paid twice as much as those who work through 50.  But I'm now thinking of the option for mods who receive better community feedback getting paid higher than the other mods. 

The following post contains the outline of all these thoughts:

https://community.narrative.or...07#15953898399098507

As you will see, this approach to the system would very tightly control when a niche owner would be able to dismiss a moderator.  Over time, the team could monitor whether that tight systemic control can be decreased or not.  After all, niche owners would suffer from dismissing good moderators, as it would disrupt the functioning of the niche and decrease revenue and content throughput.  So owner self-interest is an inherent safeguard to abuses.

I'm happy to discuss further if the @Narrative Network Team wants clarification of any of these thoughts, but I trust there is probably enough here for the team to initiate their own internal discussions on this, if any of this is new ground to them.

Original Post

Activity Stream

Malkazoid posted:

Hey!

In conversation with @Banter and @Emily Barnett and others, I came up with some thoughts that could improve moderator performance, and also give niche owners a means to address moderator underachievement effectively, within parameters controlled by the system (to prevent abuses by owners).

The thoughts revolve around moderator performance reports periodically sent to owners, and depending on the system-determined analysis for each moderator, opening up options for the owner to address issues.  They also include the concept of owners being able to have some degree of control over whether moderator rewards are somewhat performance-based, or if they all get paid equally based on the system's algorithm.  I've always assumed that algorithm would at least take into account the volume of work done by a moderator so that those mods who work through 100 posts per week would get paid twice as much as those who work through 50.  But I'm now thinking of the option for mods who receive better community feedback getting paid higher than the other mods. 

The following post contains the outline of all these thoughts:

https://community.narrative.or...07#15953898399098507

As you will see, this approach to the system would very tightly control when a niche owner would be able to dismiss a moderator.  Over time, the team could monitor whether that tight systemic control can be decreased or not.  After all, niche owners would suffer from dismissing good moderators, as it would disrupt the functioning of the niche and decrease revenue and content throughput.  So owner self-interest is an inherent safeguard to abuses.

I'm happy to discuss further if the @Narrative Network Team wants clarification of any of these thoughts, but I trust there is probably enough here for the team to initiate their own internal discussions on this, if any of this is new ground to them.

I like the idea of some monthly reports  & metrics for both moderators and niche performance.

I wondered about the moderator distribution. Especially since owners are allowed to be a moderator with out receiving any rewards ( totally agree there should be no double dipping...owner moderation should be a voluntary role). But because this it would be good to have an actual division of who did what percentage of the moderating and how they are compensated. I mean if the owner is doing 95 of the work, and the moderator is doing 5%, but receiving 100% of the mod reward fee each month, the owner should be able to intelligently spell out the disparity in work load.

@Malkazoid do you mean drop the idea of payment on volume and replace with community approval?

I think the mod that slogs through a lot of posts might not have as much time to interact as one who only does half ... as long as good post and comment quality is being maintained, volume should be considered ...

@Emily Barnett 's idea for report/metrics I think would be useful for owners who need quite a few mods - help to stay on top of what's going on.

Colleen Ryer posted:

@Malkazoid do you mean drop the idea of payment on volume and replace with community approval?

I think the mod that slogs through a lot of posts might not have as much time to interact as one who only does half ... as long as good post and comment quality is being maintained, volume should be considered ...

@Emily Barnett 's idea for report/metrics I think would be useful for owners who need quite a few mods - help to stay on top of what's going on.

Hey @Colleen Ryer - I didn't mean to suggest dropping the idea of payment on volume.  Did you check out the thread linked to above?

The idea was it might be good to allow a hybrid system, where the owner of the niche could determine a percentage of the rewards to be purely volume based, and the remainder to have quality/community approval factored in.  It seems desirable for it to be possible to include at least some consideration of how the community feels about the work of the mods?

But all of this will remain very theoretical until the functionality starts to take shape on the platform.

What was @Emily Barnett's idea about report/metrics?  Was it different from mine?  

Malkazoid posted:
Colleen Ryer posted:

@Emily Barnett 's idea for report/metrics I think would be useful for owners who need quite a few mods - help to stay on top of what's going on.

Hey @Colleen Ryer - I didn't mean to suggest dropping the idea of payment on volume.  Did you check out the thread linked to above?

The idea was it might be good to allow a hybrid system, where the owner of the niche could determine a percentage of the rewards to be purely volume based, and the remainder to have quality/community approval factored in.  It seems desirable for it to be possible to include at least some consideration of how the community feels about the work of the mods?

But all of this will remain very theoretical until the functionality starts to take shape on the platform.

What was @Emily Barnett's idea about report/metrics?  Was it different from mine?  

I proudly take no credit for ideas but my own....I was reiterating that i like your idea. I think @Colleen Ryer just mistook the sentence. 

Oops, sorry, my tracking through the posts got detoured @Malkazoid . Your suggestion re: reports/metrics. Followed by @Emily Barnett 'liking the idea.

Community member voting  decides on content quality. But some way to track how moderators are doing with keeping content and comments within the tagged subject area, or at least keeping spam/gaming out  would be useful to niche owners, aside from complaints from content contributors who think their rejected content actually fits the tag or isn't manipulative,

If there is a way to track this and I had a niche needing more than a couple of mods, I think I'd appreciate this kind of information  and being able to generate reports from it. This would also give owners more leeway in  deciding how to reward moderators 

But moderator/community interaction might be kind of hard to measure in terms of quality - and quantity might not be a useful metric. Seems owner eyeballs would be required here.

Colleen Ryer posted:

 

But moderator/community interaction might be kind of hard to measure in terms of quality - and quantity might not be a useful metric. Seems owner eyeballs would be required here.

True.  I just hope we develop ways to assess quality.

For instance if a mod rejects something from the niche that actually belonged there, and it gets appealed to the Tribunal... that should be conveniently visible for owners, so they can make up their own mind on what happened.

If the Tribunal rules the decision was incorrect, that should be tallied somewhere convenient for the owner to absorb along with other moderator metrics.

Same with flagging posts as in violation of the TOS or AUP - if those are appealed and the Tribunal disagrees, all of this should be easily absorbed by the owner.

 

Add Reply

×
×
×
×