Should the scoring system of level reviews be abolished on here?

helping builders & reviewers hone their skills

Moderators: QRS, Raymond, eTux, EssGee

Post Reply
User avatar
Ceamonks890
Newbie
Newbie
Posts: 46
Joined: 26 Sep 2012 22:25
Location: New Zealand

Should the scoring system of level reviews be abolished on here?

Post by Ceamonks890 » 14 Jun 2018 05:46

Hi everyone,

A month ago, I had decided to quit reviewing custom levels because it was starting to become too stressful to continually play them a certain way(on account of having to keep a watchful eye on general details, for the sake of a fair and objective critique on my end in regards to the scoring system, on top of the core review itself). And while I do admit that the old reviewer itch comes back from time to time while playing levels casually and ultimately feel tempted to return, the fact that I can't just upload my figurative review without giving it an almost arbitrary score, only serves to put me off that prospect.

I also bring this up, because I've come across discussions both personally & publicly that make strong arguments for why a review scoring system only serves to unfairly make certain products(or levels in this case), more worthy of your attention than others. Many forms of entertainment looked at by critics are at risk of not being experienced by many general consumers whatsoever, if they see a score anywhere lower than the highest grade imaginable and often don't bother to read the main review(s) at all for the proper context(often leading to low sales or downloads in the process). A ton of underrated gems have suffered under these redundant scoring systems(custom levels hosted on here very much included), whereas genuinely terrible titles have inexplicably prospered through bought out opinions or just general bias.

So overall, I'm probably asking for what feels like a nonsensical change on paper. But I feel in the long term, this will encourage many users on here to download levels and judge their general quality for themselves, instead of dismissing it outright over a score they personally consider too low. Who knows? Might even inspire some casual users to make accounts and become critics themselves. While on the other side of the coin, it makes the experience a lot more thrilling for veteran reviewers like me with devoting a fair amount of time, to describing the incredibly comprehensive nature of what I had recently played for the better.

But what does everyone else think? Do you agree that the scoring system should be abolished altogether on trle.net or am I just blowing things out of proportion? Looking forward to hearing all of your thoughts on this subject :)

User avatar
DJ Full
Master Chatter
Master Chatter
Posts: 6802
Joined: 02 Nov 2010 04:02

Re: Should the scoring system of level reviews be abolished on here?

Post by DJ Full » 14 Jun 2018 06:16

I mentioned it several times already and I think such change is really needed.
At this point gameplay only takes 1/4 of total score which is really outbalanced.
Much more fair and adequate would be rating with binary system: like=1, dislike=0.
MAKE WIDESHOT GREAT AGAIN

User avatar
Torry
Chatterbox
Chatterbox
Posts: 3122
Joined: 25 Aug 2001 00:38
Location: Adelaide, Australia

Re: Should the scoring system of level reviews be abolished on here?

Post by Torry » 14 Jun 2018 09:09

I do not totally disagree with your views as I have often played a level and rated it higher than the existing medium score because of the game play, fun factor and traditional raiding employed. However, some sort of scoring system is needed else there is no way to discern what efforts are to be rewarded. Now, reviewing is a very personal thing. What I like and dislike will be completely different to what another player likes or dislikes but once the reviews are collated the scores tend to even themselves out to give a viable rating. IMDB suffers just the same. What I consider a decent movie may well only have a low score so does that suggest that the movie is bad? Or that my taste in movies is bad? Read the reviews. The reviews generally explain WHY the reviewer scored it as such and then you can discern from that whether he or she is simply a wanker or so far up him/her self that they cannot see the good in anything in life. Movie critics are some such and generally are looking for specifics to rate whilst the general movie goer such as myself simply want to be entertained. So it is with Tomb Raider. Entertain me. I care not so much for stretched textures etc, what I want is a bloody good traditional raid.

Having said all that, this was all addressed some 15 years ago and nothing has changed so I do not expect it to change Ceamonks890.

Marksdad
Mod
Mod
Posts: 4867
Joined: 05 Oct 2001 17:38
Location: UK

Re: Should the scoring system of level reviews be abolished on here?

Post by Marksdad » 14 Jun 2018 11:38

Yes,this is a subject which is raised every couple of years or so.
I think it basically comes down to the fact that we have a tried and tested system that,while perhaps not perfect,is something that most people here are very comfortable with.
The 'fun factor' element is essentially reflected in the Gameplay section of the scoring system.

As Torry points out,over time the ratings for any level will even out;and will generally end up giving a balanced representation.
Provided review comments are fair,and explain why such scores were given,it shouldn't be a hardship for a reviewer to give ratings (good,bad or indifferent)

Phil
Mod
Mod
Posts: 5255
Joined: 12 Apr 2002 23:57
Location: Charlotte, NC, USA

Re: Should the scoring system of level reviews be abolished on here?

Post by Phil » 14 Jun 2018 13:47

Ceamonks890 wrote:
14 Jun 2018 05:46
A ton of underrated gems have suffered under these redundant scoring systems(custom levels hosted on here very much included), whereas genuinely terrible titles have inexplicably prospered through bought out opinions or just general bias.
I'm in general agreement with the points raised in the foregoing responses. I add my two cents only to register my strong disagreement with the basic premise quoted above. These two extremes simply don't exist in my experience. True, on occasion a worthy level will not receive the acclaim it deserves, and in the early days while we were still feeling our way along some levels certainly received higher scores than they would today, but the system has worked well in practice over the years. In my opinion it ain't broke, so there's no need to fix it.

Marksdad
Mod
Mod
Posts: 4867
Joined: 05 Oct 2001 17:38
Location: UK

Re: Should the scoring system of level reviews be abolished on here?

Post by Marksdad » 14 Jun 2018 14:47

@ceamonks,
I've just had a look at your reviewing stats.
For a total of almost 700 reviews,an average rating of 3.66 is some quite considerable distance below the average ratings of the other reviewers in the Triple Platinum category (which are all in the upper 6's or low 7's)
I looked up my own average rating (and I'm a generally forthright reviewer),and it turns out to be a comparatively generous 5.77.
Even Mulf (a reviewer not in the least afraid to express his true feelings in scores or comments) comes in higher than you,at 3.89.
You also appear to have predominantly chosen to play the lowest rated levels on the Index - which is absolutely fine. (It's good to see someone play these often ignored contributions.)
Nonetheless,perhaps (should you continue to review) you play more of the higher rated levels,you might feel a little better with yourself when it came to choosing what ratings to award them..?

User avatar
rufierto
Member
Member
Posts: 514
Joined: 03 Jul 2011 00:22
Location: Centralia, Missouri

Re: Should the scoring system of level reviews be abolished on here?

Post by rufierto » 14 Jun 2018 22:49

I think we have all at one time or another saw a reason for change, but to what is the question. I personally do not want to see it change even though I do not always think the system is fair. That being said, I still look at my old game ratings as something to beat, and I continue to feel I get better, even if the scores do not reflect that. :roll: But the system is in place, and for the most part, changing it now would be a mistake I believe. :2thumbsup:
Rescue Dutchy!!!!

User avatar
Ceamonks890
Newbie
Newbie
Posts: 46
Joined: 26 Sep 2012 22:25
Location: New Zealand

Re: Should the scoring system of level reviews be abolished on here?

Post by Ceamonks890 » 15 Jun 2018 05:32

Marksdad wrote:
14 Jun 2018 14:47
@ceamonks,
Nonetheless,perhaps (should you continue to review) you play more of the higher rated levels,you might feel a little better with yourself when it came to choosing what ratings to award them..?
Hmm, I'll have a think about it. Might indeed make it more manageable if the overall quality of a given levelset is top notch from the offset, if it means I can put more of a focus on writing without having my attention diverted elsewhere whilst playing. That's not to say I won't stop reviewing the less appealing levels offered on here altogether, but I'll probably step back from doing critiques of them as regularly(unless I somehow run out of great levels to give my opinions on in future).

I still stand by what I said, but I appreciate your outlook regardless.

User avatar
MichaelP
Master Chatter
Master Chatter
Posts: 10391
Joined: 23 Aug 2001 13:52
Location: Germany
Contact:

Re: Should the scoring system of level reviews be abolished on here?

Post by MichaelP » 15 Jun 2018 12:10

I will subscribe to the "if it ain't broke, don't fix it" comment, simply because EVERY TIME this discussion has come up in the past 15+ years, there was no convincing argument made and no alternative solution provided that had a larger group of people think that it will be a "better" approach. Whatever was suggested was always "different", but by no means provided any sensible vision of leading us to a "better" solution...

I will completely acknowledge that the system can be perceived us unfair at times - but then again, that "unfairness" is not a fault of the system, it is entirely a fault of the way the users of the system make use of it. And I do not say that to "blame" the users (reviewers) - it is just in the nature of the subjectivity of people evaluating other people's efforts.

My advice to the original "problem" articulated here: If the "burden" of scoring a level is hampering your entertainment of playing it, there is a relatively simple way out of that dilemma: play the level, have fun, enjoy writing all the comments you want to write and then simply check the current average scores of the level and make that your scores (unless of course, your perception of the level would be completely different than the average players perception).
Image

User avatar
DJ Full
Master Chatter
Master Chatter
Posts: 6802
Joined: 02 Nov 2010 04:02

Re: Should the scoring system of level reviews be abolished on here?

Post by DJ Full » 17 Jun 2018 01:58

the system can be perceived us unfair but that "unfairness" is not a fault of the system, it is entirely a fault of the way the users of the system make use of it
And that's precisely why I suggested swapping to Like/Dislike system, which is much harder to abuse. We had cases when a reviewer rated low but still liked the level. We had cases when a reviewer rated high but still disliked the level. We had cases when a reviewer wanted to rate higher/lower but was missing criterions or the existing ones weighed too much. Boring levels got into the hall of fame because of their looks, meanwhile prodigious works of subpar visuals or certain level of impolishment, like "Royal Bengal Tigers", "Silent Vill", "Serpentine Gorge", "Clara's Nightmare" and many more are waiting for over a DECADE to get admitted, despite of having unbelievably creative content contained in NO OTHER level. I met COUNTLESS streamers who downloaded games rated 9+ and were almost always disappointed, then they downloaded something rated 7+ and they utterly enjoyed. And it's very good when a player gives a chance, but think about a player who downloaded a "boring beauty", got disappointed multiple times and never tried again because he thought other 9+ are equally boring. This is even more unfair because it puts our master builders in a very bad light, think of Titia's or EssGee's levels which might be never attempted because of this process. I did a lot of work myself to convince the streamers they should lower the treshold, and play the level of any rating, and I have a proof it's correct thinking, because all of them sooner or later admit I was right. The point is it requires a lot of effort to convince people to think this way, because when a player sees 9+ he sees 9+ and when he sees 7+ he sees 7+...

I think we messed up very badly and it's very simple to fix. Like = 1, dislike = 0. And produce a percentage rating from that.
My advice to the original "problem" articulated here: If the "burden" of scoring a level is hampering your entertainment of playing it, there is a relatively simple way out of that dilemma: play the level, have fun, enjoy writing all the comments you want to write and then simply check the current average scores of the level and make that your scores
I.... I just can't believe you said that. This is just sentencing a level based on first score, and porting possible reviewer mistakes to other reviews. You don't want to fix the system but you advise to break it even more??? O.O I'm legit shocked....
MAKE WIDESHOT GREAT AGAIN

User avatar
SrDanielPonces
Member
Member
Posts: 141
Joined: 31 Mar 2015 11:35
Location: Portugal

Re: Should the scoring system of level reviews be abolished on here?

Post by SrDanielPonces » 17 Jun 2018 02:29

MichaelP wrote:
15 Jun 2018 12:10
My advice to the original "problem" articulated here: If the "burden" of scoring a level is hampering your entertainment of playing it, there is a relatively simple way out of that dilemma: play the level, have fun, enjoy writing all the comments you want to write and then simply check the current average scores of the level and make that your scores (unless of course, your perception of the level would be completely different than the average players perception).
wow that is disappointing to read.

User avatar
Ceamonks890
Newbie
Newbie
Posts: 46
Joined: 26 Sep 2012 22:25
Location: New Zealand

Re: Should the scoring system of level reviews be abolished on here?

Post by Ceamonks890 » 17 Jun 2018 03:21

MichaelP wrote:
15 Jun 2018 12:10
I will subscribe to the "if it ain't broke, don't fix it" comment, simply because EVERY TIME this discussion has come up in the past 15+ years, there was no convincing argument made and no alternative solution provided that had a larger group of people think that it will be a "better" approach. Whatever was suggested was always "different", but by no means provided any sensible vision of leading us to a "better" solution...

I will completely acknowledge that the system can be perceived us unfair at times - but then again, that "unfairness" is not a fault of the system, it is entirely a fault of the way the users of the system make use of it. And I do not say that to "blame" the users (reviewers) - it is just in the nature of the subjectivity of people evaluating other people's efforts.

My advice to the original "problem" articulated here: If the "burden" of scoring a level is hampering your entertainment of playing it, there is a relatively simple way out of that dilemma: play the level, have fun, enjoy writing all the comments you want to write and then simply check the current average scores of the level and make that your scores (unless of course, your perception of the level would be completely different than the average players perception).
I'm sorry Michael, but after having time to think over what Marksdad suggested to me, what you're advising isn't going to fix my issues with the review system on here in the long term & essentially recommends that I file in with the rest of the chorus in terms of the scoring system, finding ourselves back at square one once again(with nothing resolved on either side of this current argument). And no, I don't want to change how I critique, for the sake of a review system that is considerably flawed to the very core. If what you, Torry & Marksdad have brought up previously is true about other users inquiring for the current system to be changed for many years now, then perhaps there is something present that's putting off a large number of individuals from enjoying more levels, than they reasonably should(as implied by DJ Full).

And as such, I'm with DJ Full on this one. A like/dislike system(akin to something like YouTube), would be much more welcoming for reviewers and casual players alike, being much less likely to abuse & gives every level a fair chance in the spotlight. Everyone thus far has brought up that the current system isn't exactly fair at one point or another, so what's wrong with accepting change and taking this site in a different direction for the better? Despite what Phil says otherwise about some classic TRLE gems not being dismissed in modern times, DJ Full's evidence greatly implies differently.

So I ask again, what exactly makes the current score-based review system so much better than the offered alternative? Because the way I see it, its ultimately a win-win for everyone.

User avatar
Mr XY
Member
Member
Posts: 964
Joined: 30 Jun 2006 17:59
Location: Heidelberg, Germany

Re: Should the scoring system of level reviews be abolished on here?

Post by Mr XY » 17 Jun 2018 14:27

My question is how a general like/dislike system is fairer than a point system?

It may work on youtube because there you have a lot of likes/dislikes. Here on trle.net, so far there are only 2 levels this year that reached 10 reviews or more. My assumption is that we would end up with a lot of 100% levels because no one wants to be the bad guy and dislike a level.

And for everyone who is afraid to play a "boring beauty", check the gameplay/puzzles category. If it is significantly lower than the other 3 categories, you know what's going on. Or read the actual reviews, they are there for a reason.
Image

User avatar
Treeble
Chatterbox
Chatterbox
Posts: 3046
Joined: 08 Sep 2001 05:36
Location: Brazil
Contact:

Re: Should the scoring system of level reviews be abolished on here?

Post by Treeble » 17 Jun 2018 15:36

DJ Full wrote:
17 Jun 2018 01:58
play the level, have fun, enjoy writing all the comments you want to write and then simply check the current average scores of the level and make that your scores
I.... I just can't believe you said that. This is just sentencing a level based on first score, and porting possible reviewer mistakes to other reviews. You don't want to fix the system but you advise to break it even more??? O.O I'm legit shocked....
Not quite. Unless it's a reviewer who's always on time to be the second reviewer, which I don't think it's the case for anyone here who is not on board with the very same day one system. And quite honestly, who is taking up that advice? Nobody, that's who, it was just a throwaway comment. The OP states terrible games being uprated by bias, which also is a same old argument. Of course you are going to rate your buddy's level higher than most, but ultimately will it really affect the final average score, when everyone else isn't doing the same?

I think the bigger question all around here is: aren't we taking this too seriously? What do we get out of reviewing, anyway? A job? Our futures? It's meant to be a pastime, we all love Tomb Raider and just want endless supplies of it. And we have it, there are enough levels for two or three lifetimes even. Unless you are an aspiring game developer and making a portfolio out of your levels (which I don't think has ever been the case), and even then it might be a stretch - the reviews matter zilch in the end.

User avatar
DJ Full
Master Chatter
Master Chatter
Posts: 6802
Joined: 02 Nov 2010 04:02

Re: Should the scoring system of level reviews be abolished on here?

Post by DJ Full » 17 Jun 2018 15:58

Mr XY wrote:
17 Jun 2018 14:27
My question is how a general like/dislike system is fairer than a point system? It may work on youtube because there you have a lot of likes/dislikes. Here on trle.net, so far there are only 2 levels this year that reached 10 reviews or more. My assumption is that we would end up with a lot of 100% levels because no one wants to be the bad guy and dislike a level.
Not really. We have honest people here who will place a dislike if they dislike. I surely would.
And for everyone who is afraid to play a "boring beauty", check the gameplay/puzzles category. If it is significantly lower than the other 3 categories, you know what's going on. Or read the actual reviews, they are there for a reason.
Kevin, a player can miss a switch with obvious camera hint and downrate a level for that. Then people are advised to copy this errorscore. Like/dislike system is much more immune to this and even if it totally isn't, tell me what you prefer: a player who makes an "error" and downrates your work for stupid reasons, or a player who makes an "error" andstill chooses to appreciate what you built. And your levels are very good, they wouldn't suffer at all from this change, so I don't get why you're against.
Treeble wrote:
17 Jun 2018 15:36
I think the bigger question all around here is: aren't we taking this too seriously? What do we get out of reviewing, anyway? A job? Our futures?
When I review something I take hours of effort to make my score as good as possible. And it's not for me but for the players and builders to have the correct information. Aren't we forgetting about this?
MAKE WIDESHOT GREAT AGAIN

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest