New map rating system

[Zandril]

As we all know, Halo Customs had an update. Many new features were added nd most of them were great. However, there is one particular change that I think needs to get fixed as it is causing a problem IMO.

As the title of this thread suggests, it's the new map rating system. I'm gonna go straight to the point.

With this new map rating system, people can rate a map and then leave without an explanation. People can rate a map/gametype 1 star and then leave it at that. No words or anything to explain why.

If this system is going to stay how it is, I would at least like an explanation on why it's like that. I think this is a problem because it opens doors to trolling and grudges being acted upon.

People can rate someone's map bad because they simply don't like that person. They can rate it bad just because they think it's funny. And the person who owns the map is just confused as to why he got a bad rating.

I'm not going to lie. I started this thread because this exact thing just happened to me.

http://halocustoms.com/maps/casual-competitive-gametypes.1011/

http://halocustoms.com/maps/exin.729/

My casual competitive gametypes got a 1 star rating and I don't know why the person who did it did so. Is he someone who dislikes me? If so, way to be sore and an ahab about it. Does he actually dislike the gametypes? If so, I want an explanation because those gametypes are actually pretty solid IMO.

Same thing with my map, Exin. Same questions.

I also know for a fact that I'm not the only one who's been affected by this.

Don't get me wrong, I don't mind the option to remain anonymous when rating a map. This allows people to be more honest with their review. However, I do think it's a problem when it can be done without having to write a review.

I think it would be better if we went to the old review system or if written reviews were required and the guy can still remain anonymous. I think this new system needs a change and I know I'm not the only one who thinks that.

So. Admins? Staff? Can you guys think about changing this or at least give us an explanation?

 

[Buddy]

Finally someone who agrees with my opinion on the map rating system!

I already mentioned this in the shoutbox and like six people were against me... Let´s see if they understand the issue now.

Another map with the same problem: http://halocustoms.com/maps/electron.998/

 

[RogerDodger]

I agree with Zandril. An option to select anonymity on a written review is beneficial. However, a written review is simply irreplaceable. How are stars on a screen going to help any forger improve their map? We can't trust members of the community to responsibly leave this kind of feedback. I've reported numerous written reviews that have the same mindless hate and praise.

EDIT: Also propose that all star ratings without reviews be removed.

 

[Drizzy_Dan]

I think the new Map Rating System works well, the only tweak would be making it mandatory to leave a comment.

 

[Zandril]

Fix'd. Also, rump stants.

 

[Jebus]

Can't say it has happened to me but I can agree the system should get some sort of change, whether big or small.

 

[DeadlockDrago]

I only think it needs a change in a way people come to it, cause not many of these maps have been rated at all - good or bad. All I see it needing is some traffic to them.

 

[Player]

I agree with everything you say Zandril. I can see what the designers of halocustoms were trying to do with the rating system to make it like many other sites, but I don't believe it works when there's so few ratings on each map posted, and practically everyone knows each other. But that's an issue for another thread.

 

[Insane54]

We've messed around with both required and not required writeups, personally I lean more towards not requiring it. Reason is that the people who review and care about written reviews are a very small subset of the community we're trying to reach... by requiring a written review, you're essentially forcing it into a "forger zone", which isn't really what I'd like it to be. By allowing ratings without reviews, anyone can simply drop down a star, and due to the theoretical increase of review ratings, it converges to what people see as a better overall rating. The tradeoff is basically "force reviews, resulting in a system that isn't really relevant for non-forgers" vs. "convergence to optimality, with the option for forgers to add a review"

So, you'll occasionally get unwarranted ratings, but the increase of people rating your stuff should even it out to the star rating it deserves (given enough activity). The vast majority of websites with ratings (YouTube, for one) use this system.

 

[Auburn]

@Insane54 I wouldn't go as far as to say that a writing a review forces users into a "forger" mindset. In fact, I would even say that its important to not review maps strictly from the POV of a designer. There are several people I look to for feedback; both forgers and not. When it comes down to considering their opinions, I always lean towards the latter's because, while they may not know the standards of design like the forger does, they understand the mechanics of Halo's gameplay and know how to play the game well as a result. You could know everything about making maps in Quake, but if just apply that same knowledge in forge without understanding Halo's core mechanics, you aren't going to have a very good product. I doubt there are any forgers that don't understand the game to that degree, but you get teh point.

Back on Forge Hub, we had the idea of implementing an approval system with ratings (and reviews) that would work similarly to how infractions did back when you were admin. That being, when a user submits a rating (or review), it would be sent to the staff board to await approval before actually appearing on the public forum. I don't even know if something like that could actually be incorporated, but I thought it could work so long as the extra workload wasn't too much.

If you rate something, you should be prepared to explain why you gave the rating you did IMO. The only thing that anonymity does is show that you aren't confident in your reasoning, in which case you shouldn't be giving the rating / review in the first place.

 

[Insane54]

@Insane54 I wouldn't go as far as to say that a writing a review forces users into a "forger" mindset. In fact, I would even say that its important to not review maps strictly from the POV of a designer. There are several people I look to for feedback; both forgers and not. When it comes down to considering their opinions, I always lean towards the latter's because, while they may not know the standards of design like the forger does, they understand the mechanics of Halo's gameplay and know how to play the game well as a result. You could know everything about making maps in Quake, but if just apply that same knowledge in forge without understanding Halo's core mechanics, you aren't going to have a very good product. I doubt there are any forgers that don't understand the game to that degree, but you get teh point.

If you rate something, you should be prepared to explain why you gave the rating you did IMO. The only thing that anonymity does is show that you aren't confident in your reasoning, in which case you shouldn't be giving the rating / review in the first place.

Back on Forge Hub, we had the idea of implementing an approval system with ratings (and reviews) that would work similarly to how infractions did back when you were admin. That being, when a user submits a rating (or review), it would be sent to the staff board to await approval before actually appearing on the public forum. I don't even know if something like that could actually be incorporated, but I thought it could work so long as the extra workload wasn't too much.

Okay, maybe "forgers" is the wrong word. There is a pretty clear correlation between requirement for ratings and amount of ratings. Most people do not want to write anything. The only people that reviews benefit are the forger themselves, while the rating (if there are enough to make the ratings actually mean something) are relevant to everyone who downloads and plays maps -- the majority of people and the primary purpose of the rating system is for people to see at a glance what maps are good and what aren't. It isn't intended to be a review board, but it has the option of doing that. Given the choice of only one or the other, personally I would remove the reviews and keep the stars; they're more valuable to more people than reviews are.

The idea is to decrease, not increase, the amount of ratings going on. Ideally, very few ratings will have attached reviews. If you have 100 reviews on a map, outliers (unwarranted 1 and 5 star ratings) become negligible, and the 'average' brings out approximately how good a map is. For the vast majority of HaloCustoms' users, this is exactly the functionality they want -- at a glance, a general idea of the quality of the map they're looking at. On the other hand, requiring ratings ends up with very few star ratings, generally meaning you're reliant on only one or two good reviews. Currently, look at the maps section. Do any of the star reviews mean anything to you? They shouldn't. People don't really review bad maps, making it all high starred maps. While its helpful for the person posting, it's not very helpful to anyone else. By allowing everyone to say "3 stars!" without needing to explain it, it increases the amount of ratings, making that number relevant. Feedback, of course, is very much appreciated, but if feedback were required I'd probably remove the stars, since they're unnecessary at that point.

To give you an idea of what I mean: imagine if you had to post a thoughtful review in order to "thumbs up" or "thumbs down" on YouTube. Imagine if you had to do that to "Like" something on Facebook! The way these things work are convergence -- lots of people posting unwarranted reviews (with a few real ones) ends up in roughly the correct place, as numbers increase.

 

[RogerDodger]

"Most people do not want to write anything."

So, most people are lazy. Fine. If they actually had an interest in the map, they would stop by to write a few measly words. "Good job!" "This map sucks." Something. I've seen them around, too.


"The only people that reviews benefit are the forger themselves, while the rating (if there are enough to make the ratings actually mean something) are relevant to everyone who downloads and plays maps -- the majority of people and the primary purpose of the rating system is for people to see at a glance what maps are good and what aren't."
"While [reviews are] helpful for the person posting, it's not very helpful to anyone else."

So the logic here is, "let's trust the opinions of a bunch of random people, rather than one or a few people who can articulate what they're talking about". What's the point of the reviewer in this scenario? It's not just forging help. We leave specific details about how the map plays, and what players can expect. They are for people who want to make educated downloading decisions. Stars are just a shot in the dark.

A quote from Roger Ebert:
"One thing I try to do is provide an accurate account of what you will see, and how I feel about it. I cannot speak for you. Any worthwhile review is subjective. If we completely disagree, my words might nevertheless be useful or provocative. If you disagree with what I write, be my guest. If you disagree with how many stars I gave it, you can mail your opinion to where the sun don't shine."


"The idea is to decrease, not increase, the amount of ratings going on."

Forgers will naturally have less usable feedback because of this. Why type anything (even in the Discussion tab) when you can express yourself with a single click?


"If you have 100 reviews on a map, outliers (unwarranted 1 and 5 star ratings) become negligible, and the 'average' brings out approximately how good a map is. For the vast majority of HaloCustoms' users, this is exactly the functionality they want -- at a glance, a general idea of the quality of the map they're looking at."
"To give you an idea of what I mean: imagine if you had to post a thoughtful review in order to "thumbs up" or "thumbs down" on YouTube. Imagine if you had to do that to "Like" something on Facebook! The way these things work are convergence -- lots of people posting unwarranted reviews (with a few real ones) ends up in roughly the correct place, as numbers increase."

Based on the active population of the website and Halo in general, we won't have the volume to blast outliers into obscurity. For example, (1 + 3 + 3 + 3) / 4 = 2.5 ... is that at all fair? Sure, it won't always be in that 1:3 ratio. I'm not worried about the trolls, but the masses of people who don't know how Halo works, determining the quality of maps.

Also, maps can't be compared to these media sources. Media is one-use disposable, fast, unchanging, and (usually) conceived without much effort. Good maps are the product of a dynamic and intensive process, and can be used for a long time.


"Currently, look at the maps section. Do any of the star reviews mean anything to you? They shouldn't. People don't really review bad maps, making it all high starred maps."

People don't have the balls to tell someone their map is bad. Also, they wouldn't want to play it to find out themselves. That's where an anonymous written review would be great. Hide behind the silkscreen, fine. As long as you have good reasoning, it's okay to remain anonymous.


This is how I see people in regards to map ratings/feedback:
(And yes, I spelled "knowledgeable" incorrectly. Sue me.)

Spoiler

Concluding statement:
If people do not have the knowledge or effort to post a review, then they shouldn't have a voice in rating the map.

 

[Buddy]

You´re right, Insane. But the problem with your point is that there are Not like 20 anonymous ratings on a map, but like 2; and two don´t represent the quality of the map. It´s a question of how many people even want to review a map.

I don´t know, I think maps should be a more primary thing on HC. They should be more highlighted. Without good maps the custom lobbies are not good, too. The maps are the roots of the big stock of customs...

 

[Greenh2]

Yeah, I think that needs a change. I actually have an idea for a rating system. Ok, there will be two versions of ratings. Where the ratings are at when you first go into the "maps" section, there will be two lines of stars. One labeled "People ratings" while the other is called "Review ratings". It'll separate the two ratings from each other.

 

[Player]

But this isnt* Youtube popularity level... Maybe it will after a few years or maybe never. I don't think we're ready for this type of review system unless, like buddy said, maps are made a more primary feature on the site. Home page even? That might be too much publicity, but I don't know. I could be wrong.

 

[Auburn]

Feedback, of course, is very much appreciated, but if feedback were required I'd probably remove the stars, since they're unnecessary at that point.

if feedback were required I'd probably remove the stars

I'd probably remove the stars

remove the stars

Yes please.

 

[Insane54]

Yes please.

Which would make the entire thing useless for anything but a map repository. This is what happened to ForgeHub, it became a map repository rather than a convenient way to find good maps. Pretty soon, people realized "hey, there's better ways to find good maps! We can go to THFE and they'll tell us what's good!", and it's true. THFE does a better job of telling you what to play than a map repository does, by far, AND its easier to digest, AND you can see the map before you download it. Suddenly, the only people posting are forgers, and the only comments are by other forgers giving feedback, eventually dying out as people start leaving due to lack of activity. ForgeHub's critical flaw was being unable to adapt and unwilling to change. In my case, people have argued that my approach falls too much towards rapid change, but you guys are here, so what does that tell you?

So the question is, is the Maps feature meant for solely map creators and dedicated reviewers... or is it meant to help people set up customs conveniently? Does Maps compete with or complement YouTube channels like THFE?

Currently, the only reason the review feature is in there at all is as a kind of "tip of the hat" towards map creators, because sometimes people do have genuine feedback, and its really nice to get feedback on your work. However, becoming yet another map repository is the last place I want to go with this website.

 

[Zandril]

So the question is, is the Maps feature meant for solely map creators and dedicated reviewers... or is it meant to help people set up customs conveniently?

But if good maps are being rated badly, then customs would also take a hit. Wouldn't it be easier to find the good maps in map listing if they had the ratings they deserve. If people can rate a map bad because of the reasons I mentioned in the OP, then the good maps would have a bad cover because of the bad rating. People see the bad rating and will probably ignore the map because of it.

It would be more convenient for customs if the good maps are being highlighted with proper ratings instead of being shunted down by unexplained bad ratings.

 

[Insane54]

But if good maps are being rated badly, then customs would also take a hit. Wouldn't it be easier to find the good maps in map listing if they had the ratings they deserve. If people can rate a map bad because of the reasons I mentioned in the OP, then the good maps would have a bad cover because of the bad rating. People see the bad rating and will probably ignore the map because of it.

It would be more convenient for customs if the good maps are being highlighted with proper ratings instead of being shunted down by unexplained bad ratings.

Check out the map database right now and tell me if you can pick out a good map from a bad map based on star rating. No? That's what it looks like when reviews are required; since they are, only people in a very small circle actually bother to review it, ending up in this endless cycle of 4 and 5-star reviews. With a large amount of 'general' reviews, you (theoretically) reach a point where star rating reflects popular opinion -- when 100 people vote '4 stars', the few that thought it was funny to vote 1 star become negligible. We're very clear about how star rating is purely by popular opinion, not relative quality. For an assurance on quality maps, there's always Staff Picks and THFE's maps, all of which are featured specially -- to me, a high star rating should mean "the average user likes this map", which I think really gives it some relevancy to the majority of our users.

To quell some fears about getting one troll rating and being shot down, I believe the system shouldn't display the rating until it has a few ratings to base of off. Therefore, while someone could give a 1-star review (whether warranted or not), it shouldn't display until at least a couple of votes are in. I'm not 100% sure if this is currently implemented or half-way done, I'll be sure to test it when I get a chance.

 

[Auburn]

Which would make the entire thing useless for anything but a map repository. This is what happened to ForgeHub, it became a map repository rather than a convenient way to find good maps. Pretty soon, people realized "hey, there's better ways to find good maps! We can go to THFE and they'll tell us what's good!", and it's true. THFE does a better job of telling you what to play than a map repository does, by far, AND its easier to digest, AND you can see the map before you download it. Suddenly, the only people posting are forgers, and the only comments are by other forgers giving feedback, eventually dying out as people start leaving due to lack of activity. ForgeHub's critical flaw was being unable to adapt and unwilling to change. In my case, people have argued that my approach falls too much towards rapid change, but you guys are here, so what does that tell you?

So the question is, is the Maps feature meant for solely map creators and dedicated reviewers... or is it meant to help people set up customs conveniently? Does Maps compete with or complement YouTube channels like THFE?

Currently, the only reason the review feature is in there at all is as a kind of "tip of the hat" towards map creators, because sometimes people do have genuine feedback, and its really nice to get feedback on your work. However, becoming yet another map repository is the last place I want to go with this website.

When it comes to rating systems, the majority doesn't care to understand the "why" behind scores. They just take whats in front of them because its there. Who can blame them, though? They see a hierarchy, so they're naturally going to reach for the top. How many users stop to check the maps in the middle, though? What's to say the maps above them are better? What's to say the maps below them are worse? People don't stop to ask those questions and instead, just keeping pickin' from the top because its convenient. Yes, repositories aren't good, but rating systems are just as bad IMO. I digress, though. I understand why you use ratings and I'm really not looking to change your mind on the subject lol.

Concluding statement:
If people do not have the knowledge or effort to post a review, then they shouldn't have a voice in rating the map.

I can't say i agree with this either. Everybody should feel comfortable writing a review whether they have the "knowledge" or not. I mean, you don't need extensive knowledge to tell somebody what you liked or disliked about their map. I do think that when people do make ratings or reviews, however, that they should take responsibility for doing so; which brings me back to the anonymity case. if they're making a rating anonymously, chances are they aren't confident in their reasoning for giving said rating. If that's the case, then yes, they shouldn't be giving one in the first place IMO.