Should the scoring system of level reviews be abolished on here?

helping builders & reviewers hone their skills

Moderators: eTux, QRS, Raymond, EssGee

Post Reply
User avatar
Teone
Senior Member
Senior Member
Posts: 1024
Joined: 16 Mar 2017 16:59
Location: Italy

Re: Should the scoring system of level reviews be abolished on here?

Post by Teone » 20 Jun 2018 15:24

Marksdad wrote:
20 Jun 2018 14:57
:blush: Put me in a corner,there!
Ok,here's the story behind that..
Oh, I didn't know this story! Sorry, get out of that corner and don't be shame. :D
This story teach us that maybe is easy to create an algorithm for likes, while it would be not so easy to create an algorithm for fake-verbose-reviews. In my opinion, the number of downloads and the number of reviews say more than a like.
Image

Dick
Chatterbox
Chatterbox
Posts: 2159
Joined: 22 Mar 2002 16:01
Location: England

Re: Should the scoring system of level reviews be abolished on here?

Post by Dick » 20 Jun 2018 16:20

Teone wrote:
20 Jun 2018 14:10


...but sometime also sorting by numbers of download.
You'll miss out on old classics sorting by download because the download counter only went up at the start of 2012. In fact it's the levels from 2001 that got the most downloads.

re: the 'likes' system: schoolkids and their pesky IT projects notwithstanding, it's all a bit suspect. Enthusiastic fans sometimes cause big jumps by pressing the button over and over. It also reduces the opinion of a player to a single data point. The usual online star-rating system is not much better.


Better the detailed feedback we get now which as Jess Joho of Motherboard said in her article on trle:
The 2,000 members on the site's forum have submitted around 50,000 reviews of said custom levels, accumulating a robust rating system that would give Metacritic a run for its money.

User avatar
Tolle87
Member
Member
Posts: 959
Joined: 12 Oct 2012 11:31
Location: Belgrade, Serbia
Contact:

Re: Should the scoring system of level reviews be abolished on here?

Post by Tolle87 » 20 Jun 2018 19:16

You're probably aware that the builders can like their own level as many times as they want. So much for the credibility of likes :)
A man who introduced me to Lara back in 1996, more than a friend, my second father....
Dusan "Dule" Petrovic (1965-2018)

User avatar
MichaelP
Master Chatter
Master Chatter
Posts: 10398
Joined: 23 Aug 2001 13:52
Location: Germany
Contact:

Re: Should the scoring system of level reviews be abolished on here?

Post by MichaelP » 20 Jun 2018 22:22

Let me Thank you all for an interesting and factual discussion. We have had them before and they were sometimes not so "pretty" ;)
Really appreciate that Jake started this thread and that what followed was a generally good exchange of positions on the subject.
It will probably not be a huge surprise that I still do not see the "drama" around the sytem being "so flawed that it will lead to the downfall of the community very soon", while with a different and more innovative (?) approach we would flourish and reach new heights of activity and engagement...

... BUT ... I also don't want to be ignorant to new ideas.

My struggle is a bit... what exactly ARE these new ideas?

I have read here about:
Idea 1: Drop ratings altogether and only rely on comments (Jake) and
Idea 2: Replace ratings by a simple Like / don't like choice (DJ)
Idea 3: Re-arrange / regroup the rating categories one way or another (Tolle and others?)

With all due respect, none of these ideas are the most innovative ideas at first glance, so let me challenge those of you who want to push for a change to think through a bit more about what it should really look like instead, how should it be represented on this site in the end and why you believe it will then be less "flawed" than the current system. If you present a (sort of) holistic concept that I can understand and grasp, then I am quite willing to introduce it as an alternative solution to the current system that people can either choose one of the two when they review a level or use both systems if they wish to do so.

For example, related to the above "ideas":
Idea 1 - would be real easy to do "technically" of course and then essentially eliminates any of the rating and ranking related pages - no Hall of Fame, no Levels of the Month, no Top 100 anymore. Just a laundry list of levels which you can sort by date of release, review count, maybe download count and otherwise you just need to do a lot of review reading to choose whether you want to play a level or not... Not sure how that is "innovative"?

Idea 2 - actually also not so hard to do technically - for example I could simply translate all scores of say 7+ into a "like" and all below into a "dislike" to retain some of the historical data we have from almost 60000 Reviews. Doing so would then put Himalayan Mysteries, Mist of Avalon, Beneath the Forbidden City, Sanctuary of Water Ice and Fire and The Last Crusade into the Top 5, simply because they are great levels AND had many Reviews (= Likes), while Hypersquare (for now) would end up somewhere around 2000th rank with currently 8 Likes... Hmm - again - how would that be innovative?
OR - we leave history alone and look forward only, simply giving an option when somebody submits a review to either score the categories as before OR choose a simple like/dislike option or both - whichever they prefer. Then of course - what do we do with this mix in terms of Hall of Fame, Levels of the Month, any kind of ranking and sequencing mechanism??? Oh and not to forget - aspide actually has something similar in place today with a "pick out of 5 stars" rating. Do you think this is working well??

Idea 3 - sure - we can get very creative on mixing categories in different ways and even could define a logic of some sort that translates/transfers the historical data. But what should that logic exactly look like? And what would be the arguments to claim that the new one is better than the old?

Bottomline, I want to be open for critique, but the critique I have read in this thread so far is rather more "opinionated" and not very "substantiated" and the proposals for change are not very well thought through to the end in my opinion... Happy to "work" on this more with whoever cares and has the passion for it. :D
Image

User avatar
rufierto
Member
Member
Posts: 515
Joined: 03 Jul 2011 00:22
Location: Centralia, Missouri

Re: Should the scoring system of level reviews be abolished on here?

Post by rufierto » 20 Jun 2018 22:54

I have been critical myself when it came time to accept my reviews, but at this point, changing the system does not make any sense. Yes the conversation comes around, usually when someone like me gets ratings less than they expected, and I have come to the conclusion I was out of line. The system will never be perfect, someone will always be getting their feathers ruffled. But trying to switch over to any other system will be a logistical nightmare, and change how things sit now. Any new system needs to be on a new site. This has been going on for a long time. We are lucky to have this site, and so many people willing to rate the levels, most I believe take pride in that, deservedly so. There is a lot of commitment to the system here, and it should continue.....Just my 2 cents. :hammer:
Rescue Dutchy!!!!

User avatar
Jorge22
Master Chatter
Master Chatter
Posts: 9543
Joined: 21 May 2003 04:26
Location: Portugal (Porto area)

Re: Should the scoring system of level reviews be abolished on here?

Post by Jorge22 » 21 Jun 2018 01:42

Ditto (not a surprise either).
Image

User avatar
MichaelP
Master Chatter
Master Chatter
Posts: 10398
Joined: 23 Aug 2001 13:52
Location: Germany
Contact:

Re: Should the scoring system of level reviews be abolished on here?

Post by MichaelP » 21 Jun 2018 18:30

for statistics fans.... :D

Image
Image

Phil
Mod
Mod
Posts: 5282
Joined: 12 Apr 2002 23:57
Location: Charlotte, NC, USA

Re: Should the scoring system of level reviews be abolished on here?

Post by Phil » 21 Jun 2018 19:11

Some interesting spikes (probably attributable to freshness - that first spike - and to special events such as BtB). There's a steady decline until 2014, then the numbers start picking back up. If we're looking for signs of a dying community, this graphic doesn't support that argument.

User avatar
rufierto
Member
Member
Posts: 515
Joined: 03 Jul 2011 00:22
Location: Centralia, Missouri

Re: Should the scoring system of level reviews be abolished on here?

Post by rufierto » 22 Jun 2018 00:56

Overall, nothing has changed since 2004, still rocking!!
Rescue Dutchy!!!!

User avatar
billie2001
Newbie
Newbie
Posts: 64
Joined: 30 Jul 2016 02:38
Location: Greece

Re: Should the scoring system of level reviews be abolished on here?

Post by billie2001 » 22 Jun 2018 14:45

MichaelP wrote:
20 Jun 2018 22:22
Idea 3 - sure - we can get very creative on mixing categories in different ways and even could define a logic of some sort that translates/transfers the historical data. But what should that logic exactly look like? And what would be the arguments to claim that the new one is better than the old?
The argument here is that it would attract more reviewers, and we do need them if we expect the reviewing/rating system to be fair.
As it is now, most of the reviewers are builders or expert players. They write and rate according to technical facts that most players don't know a thing about, discouraging the less savvy from becoming reviewers.
I know if the overall visuals are good or bad, but I don't know why, meaning I cannot rate the visuals in separate categories.
How can I rate the secrets I didn't find? They are part of the gameplay. If I cared enough to search for them but didn't find them, I can rate the gameplay accordingly. Enemies, their variety and placement in the game is also a matter of gameplay. There are masterpieces that have no enemies at all. And so on...
And the most important thing is completely left out in this rating system: the enjoyment factor. As we read in so many reviews, the reviewer enjoyed the game quite a lot, despite its (technical) faults that forced him/her to give lower ratings.
The "like" button is a good thing and it could represent the enjoyment factor if it worked correctly: Registered users (so it can be pressed only once) and be able to see their nicknames. As it is now, it is completely useless.

Although off topic, I would like to suggest a difficulty tag (easy/medium/challenging/very challenging) that some older levels already have. It would give beginners an idea if the level is within their skills and help hard-core players to avoid the so called "boring" levels. If we could search levels by this tag, it would be even better.

Marksdad
Mod
Mod
Posts: 4869
Joined: 05 Oct 2001 17:38
Location: UK

Re: Should the scoring system of level reviews be abolished on here?

Post by Marksdad » 22 Jun 2018 16:03

They write and rate according to technical facts that most players don't know a thing about, discouraging the less savvy from becoming reviewers.
With respect,I don't agree with this. This system has been in place for the better part of eighteen years.It never discouraged people from becoming reviewers back then,nor at any time since (we see new reviewers popping up with gratifying frequency).
What has happened,though,is the formation of a sort of loose duology comprised of 'informed builder reviewers' and 'reviewers who just enjoy the experience of playing'. The existence of both is invaluable,imo;as both groups not only come to the game with potentially different priorities;but will reflect these differences in their reviews and scores;making them more representative as a consequence. Both these groups seem to be able to co-exist quite happily.

Your other observations I don't dispute at all :) The rating for secrets,for example,has always been a little flawed. I mean,how can you rate something which is specifically intended to not be easily identified? :? :)

Phil
Mod
Mod
Posts: 5282
Joined: 12 Apr 2002 23:57
Location: Charlotte, NC, USA

Re: Should the scoring system of level reviews be abolished on here?

Post by Phil » 22 Jun 2018 17:54

billie2001 wrote:
22 Jun 2018 14:45
I know if the overall visuals are good or bad, but I don't know why, meaning I cannot rate the visuals in separate categories.
As Justice Potter once famously said in a porn case: "I may not be able to define it [pornography], but I know it when I see it."

I'm one of those jaded reviewers who can enjoy the eye-pleasing qualities of levels without knowing the first thing about textures or other technical aspects of building. Even though there's not a category for "enjoyment factor" or "visual appeal," I have no problem using the existing system to register my approval or disapproval with sufficient accuracy.

User avatar
MichaelP
Master Chatter
Master Chatter
Posts: 10398
Joined: 23 Aug 2001 13:52
Location: Germany
Contact:

Re: Should the scoring system of level reviews be abolished on here?

Post by MichaelP » 22 Jun 2018 18:59

Phil wrote:
22 Jun 2018 17:54
... I have no problem using the existing system to register my approval or disapproval with sufficient accuracy...
Exactly!! I see the "issue" rather more with reviewers who somehow feel compelled to "stick to some perceived rules" vs interpreting them in a way that suits their way of how they want to review... Whichever categorization you choose in the end, it will resonate perfectly with some and not at all with others...

Some people seem to be very meticulous and score in a way that that they say, hey is the category is called Enemies, Objects and Secrets, then I have to give about 33% of my score for enemies, 33% for objects, 33% for secrets...
Others go about it more by saying ok, if I perceive Enemies and Objects used and Secrets placed as "perfect" then it is a 10 and if I encounter anything there that annoys me then I start deducting points from the 10...
And others may still come up with other means of deriving a score, maybe more by a general gut feeling....

The point is - you need to find your own way and rate/score consistently within that mechanism and everything will be fine... there is no right or wrong way. If you want to score your "enjoyment" you have 4 categories to do so. I for one do indeed include this in my evaluation of Gameplay & puzzles, but nobody says that this would be the right and only way. You could also choose to apply a more technical evaluation of the categories and then choose to add a point for each if you generally liked the level a lot... or whatever else suits your personal go at reviewing....

For example, the Levelbase has chosen to separate Gameplay and Puzzles into two separate categories and also Atmosphere and Sound into two separate categories. Personally, I don't like that, but if I would review there, I could also adapt to that without much trouble...
Image

User avatar
Lara_Fox_Croft
Newbie
Newbie
Posts: 57
Joined: 11 Jun 2012 19:30

Re: Should the scoring system of level reviews be abolished on here?

Post by Lara_Fox_Croft » 22 Jun 2018 21:45

Wow, this is quite a hard topic here. :argue:

IMO everyone can rate levels like he wants, like said Mr Prager, there are categories, but you don't have to stick perfectly to them.
I use to review in a special way, like some others do, taking every category and rating in each category what I remember in the level, I start with 10 10 10 10, and then I see how much work/how much fun/how beautiful the level is, and then I chose to down the rate, or not. :bandana:

For some levels, I tried to be as objective as I could, when I saw non-seamless textures, wallpaper textures etc etc but I had fun in the level, should I put a 10 in lighting and texture? Sure I won't, because a level can be fun and pretty ugly, and a good looking level but too hard/too tricky/too boring won't have a 10 in each category, we can't forget that we builder for players, and ourselves, I won't release a level that I think is boring, what would be the point? I use to think that everyone think the same way, so when I review a level I'm trying to understand why a player made an hard fight on 1 sector large platforme, or a timerun with a lot of enemies around, was that funny or tricky?
to sum up, everyone see the world from his/her window, and we review as we can/want.
Changing the system could not change it. :beatnik:

If I want to review, I review using the site rules, that's all, I don't review in other websites, maybe because I don't want to review 3-4 times the same level, but if I would do it, I would change the way to review using " user condition"/ category rules.
:hammer:

A like/dislike option could be interesting if only we were millions here...I guess we are not. So that would be pretty bad for some levels. Plus we have " like " button here...Why should we add a dislike one?
If you think this sould be fairer, why do we have to rate a level? only a review without rates would be enough. :hammer: :bomb:

About hall of fame...Well it took me years to have one level in it, that's normal, your level must be good enough to be in there, why bad builders could have levels in there? Would it be fair to some other builders? :bobby:




Well, sorry for by pretty bad english (again) :bigcry: :baby:


Regards :D :angel:

User avatar
Mytly
Keen Eye of the Week
Keen Eye of the Week
Posts: 43143
Joined: 23 Aug 2009 23:48
Location: India

Re: Should the scoring system of level reviews be abolished on here?

Post by Mytly » 23 Jun 2018 18:16

MichaelP wrote:
22 Jun 2018 18:59
The point is - you need to find your own way and rate/score consistently within that mechanism and everything will be fine... there is no right or wrong way.
This seems so obvious to me that I'm genuinely surprised that many people don't see it this way. :? The Review Guidelines are guidelines, after all, not rules.

Perhaps it would help to emphasize this point on the review page - that is, make it clear that the guidelines are only meant to guide the reviewers, not meant to be rigid rules they must adhere to.
billie2001 wrote:
22 Jun 2018 14:45
Although off topic, I would like to suggest a difficulty tag (easy/medium/challenging/very challenging) that some older levels already have. It would give beginners an idea if the level is within their skills and help hard-core players to avoid the so called "boring" levels. If we could search levels by this tag, it would be even better.
This would indeed be useful. A consistent difficulty tag in combination with reviews and ratings would give a good idea of what the level is like. For example, a level tagged 'difficult' and rated above 9 is bound to be a very different kind of level from one tagged 'easy' and rated above 9. Reviewers should also keep in mind the difficulty tag before rating a level - for example, if a level is meant to be easy (as Xmas or Easter levels often are), then it's not fair to downrate it for being easy - and vice versa.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest