Should the scoring system of level reviews be abolished on here?

[Jorge22]

I see your point.

Tell you what: I could divide the ratings into two categories - gameplay elements (which can include God knows how much) and atmosphere elements (including the sounds, the architecture, the lighting, all that and a lot more).

I think that could make some sense. Still, I wouldn't recommend it.

[MichaelP]

Tolle, yes I see your point as well.

And no, I do NOT think the current system is the best possible - I can envision many similar systems that are equally good or bad - depending on how you look at it. What I keep arguing is that I have not been convinced enough to change to something that is then going to be "better" (whatever that even means...)

Your example would indeed be a bit concerning (and frankly that concern is entirely independent from recent developments around the editor etc), IF reviewers would actually be scoring as strictly as your example implies. However - that is actually not the case. If you look at scores mopre typically given, you will very, very rarely find that people go to these relative extremes. It is quite rare to see the category scores be further apart from each other than 2-3 points. So, more typically what you would see is:

Case a) Super gameplay - not so good "atmosphere etc elements" leads to high gameplay score and slightly lower other scores, so rather a 10/5/5/5, you would see a 10/8/7/7 = 8.0

Case b) Not so good gameply, but totally stunning looks, the same the other way round - stunning Looks will lead to relatively higher score for gameplay than same "quality" gameplay with bad looks, so more like a 7/10/9/10 = 9.0

And it all remains in the control of the reviewer anyway. If I would at first - category by category - score a level 10/5/5/5, but then think that an average score of 6.25 really does not do it proper justice, then I have the choice as a reviewer to rate it 10/7/7/7 and still comment on what I did not like so much in the 3 categories. Or vice versa of course.

So yes, I get it - we do have this 3 categories vs 1 "problem". But you can also look at it the other way around:
As a builder - how much effort and passion and time do you invest into gameplay vs [enemies/objects / atmo / sound/ cameras / lighting / textures - altogether]?
I would argue that is NOT a 50/50 split, so isn't it justified to evaluate your work a little bit "off balance" here?

No right or wrong answers on this one, of course. Just trying to shed a bit of a different light on a valid point made

[billie2001]

I totally agree with Tolle. Two categories would be enough (gameplay and visuals). The specifics can be explained in the review.
But it would be difficult and rather unfair to translate the ratings of the old system to the proposed new one because the categories as they are now are a mixed bag.
1. Gameplay & Puzzles: this one is pretty clear.
2. Enemies, objects & secrets: IMO enemies and secrets are part of the gameplay. Objects (and their placement) are more in the visual department. How will you split the rating?
3. Atmosphere, sound and cameras: A mixed bag also. Cameras are part of the gameplay, otherwise they are useless. The rest have nothing to do with the gameplay. Again, how will you split the rating?
4. Lighting and textures: this one is also pretty clear.
But anyway, we don't have 3 categories for visuals. We have one for gameplay, one for visuals and 2 categories that contain both elements.

[Tolle87]

What I keep arguing is that I have not been convinced enough to change to something that is then going to be "better" (whatever that even means...)

The closest way to explain what I mean by "better" is this - Review grading (of one reviewer) should match the impression grading (of the same reviewer). As it is now, you can get into the situation in which you like one level way more than another, but the review system actually forces you to rate them (overall) equally or even the opposite (I know you suggested the solution for that, more on that below).

Your example would indeed be a bit concerning (and frankly that concern is entirely independent from recent developments around the editor etc), IF reviewers would actually be scoring as strictly as your example implies. However - that is actually not the case. If you look at scores mopre typically given, you will very, very rarely find that people go to these relative extremes. It is quite rare to see the category scores be further apart from each other than 2-3 points. So, more typically what you would see is:

Case a) Super gameplay - not so good "atmosphere etc elements" leads to high gameplay score and slightly lower other scores, so rather a 10/5/5/5, you would see a 10/8/7/7 = 8.0

Case b) Not so good gameply, but totally stunning looks, the same the other way round - stunning Looks will lead to relatively higher score for gameplay than same "quality" gameplay with bad looks, so more like a 7/10/9/10 = 9.0

Your example still shows my point. The overall diffence of 1 full point in favor of level with 3 full points weaker gameplay. That might be okay with extremely pretty vs extremely ugly levels, but here....

And it all remains in the control of the reviewer anyway. If I would at first - category by category - score a level 10/5/5/5, but then think that an average score of 6.25 really does not do it proper justice, then I have the choice as a reviewer to rate it 10/7/7/7

Yes, exactly, you can do that. In other words, you have to change your written ratings by two full points in three categories from your actual honest impression for those categories, just so you can artificially make the overall rating to match your overall impression. That doesn't sound the alarm?

As a builder - how much effort and passion and time do you invest into gameplay vs [enemies/objects / atmo / sound/ cameras / lighting / textures - altogether]?
I would argue that is NOT a 50/50 split, so isn't it justified to evaluate your work a little bit "off balance" here?

I'm not sure about the invested time, it depends if you count the time when I'm away but still thinking about gameplay ideas and "build" puzzles inside my head (which I can't do with visuals). But I still don't see why the mere number of invested hours should matter. You know that one simple room can take hours and hours to build but the players can just rush through it without even noticing it, while on the other hand the elaborate puzzle can be built in a relatively short time and still amaze the player. The players, as you said, evaluate my work and not my work in correlation with the invested time.

No right or wrong answers on this one, of course. Just trying to shed a bit of a different light on a valid point made

I got that

I totally agree with Tolle. Two categories would be enough (gameplay and visuals).

I didn't exactly suggest two categories. I'm generally suggesting the increasement of gameplay influencing the ratings by dividing it into three (or however we agree) categories. That would also have additional value of making more distinction between different types of levels (yes, I know the reviewer can explain all that in the review, but if that was enough we wouldn't even need the marks).

[Jorge22]

Yes. For a while there I almost doubted myself. But I added I didn't recommend it.

Anyway, I'd just like to add I never have to adapt my written review to my ratings nor the opposite. They simply come together naturally.

That and that, pretty please, if you ever change the way ratings are given don't even think of changing what's already been done before. Pretty please indeed as reviewers wouldn't recognize themselves in those changes. I know I wouldn't. I can't even grasp how anybody would ever do that in a reasonable manner.

[MichaelP]

Interesting - billie's post already shows how perception of what is "gameplay" vs what is "visuals" can vary quite a bit from person to person

Now, out of curiosity, and neglecting billie's perspective for a moment, here are a few stats again: If I go and do a simple formula and give the gameplay and puzzles score of each review a "weight" of 3x to "balance" it with the 3 "visuals categories" (ie I cacluate the Review score as [ (3x G&P + EOS + ASC + LT) / 6 ], what happens then?

#1 There are actually only 2 Levels out of 3000! where the Overall score difference would be > 1 Point:
The Jade Moon Demo by Mountain Dew Nut - goes down from 4.43 to 3.23 and
Night Castle by Oxy - goes down from 6.46 to 5.42
Both examples have "limited" gameplay, but solid visuals apparently...

#2 The other way round there is only one Level that hits a 0.5 Point difference, all others are less:
Level 0 - Tutorial by NepT1 - goes up from 5.75 to 6.25
It clearly has very solid gameplay and not so exciting visuals, but still the difference is not that big.

#3 What would the Top 20 look like? (note that I am working of a Review extract from a few days back, so some of the latest ones are not reflected here)

With the current scoring and ignoring the calculation that eliminates lowest and highest scores, the Top 20 of all time would be:

3037 - Ice Age 3 - 9.95
3032 - Hypersquare 1+2 - 9.94
1591 - Experiment 3 Budapest - 9.93
1610 - Underworld UB4 - 9.88
2500 - Mists of Avalon - 9.87
1124 - Underworld UB1 - 9.86
779 - Sanctuary of Water, Ice and Fire - 9.86
1793 - Himalayan Mysteries - 9.86
1315 - Underworld UB3 - 9.85
1182 - Underworld UB2 - 9.83
1565 - Neon God - 9.83
2589 - King Arthur Project - 9.83
1378 - Jungle Ruins 3 - 9.83
2388 - Hidden Garden NG - 9.82
2586 - Coyote Creek 2 - 9.82
2727 - Hypersquare - 9.81
2697 - Tomb Raider Redemption - 9.81
2441 - Jerusalem Project - 9.81
2949 - Tomb Raider Requiem - 9.80
1115 - TRA Techno Egyptians - 9.80

With the 3x factor for GP, this turns into:

3037 - Ice Age 3 - 9.95 > 9.97 (0.02)
1591 - Experiment 3 Budapest - 9.93 > 9.92 (-0.01)
3032 - Hypersquare 1+2 - 9.94 > 9.88 (-0.06)
1124 - Underworld UB1 - 9.86 > 9.87 (0.01)
1793 - Himalayan Mysteries - 9.86 > 9.83 (-0.03)
1182 - Underworld UB2 - 9.83 > 9.82 (-0.01)
1610 - Underworld UB4 - 9.88 > 9.81 (-0.07)
779 - Sanctuary of Water, Ice and Fire - 9.86 > 9.81 (-0.05)
1315 - Underworld UB3 - 9.85 > 9.80 (-0.05)
2589 - King Arthur Project - 9.83 > 9.80 (-0.03)
1836 - The Quest of Gold - 9.78 > 9.80 (0.02)
1115 - TRA Techno Egyptians - 9.80 > 9.79 (-0.01)
2500 - Mists of Avalon - 9.87 > 9.79 (-0.08)
2697 - Tomb Raider Redemption - 9.81 > 9.76 (-0.05)
1565 - Neon God - 9.83 > 9.75 (-0.08)
1378 - Jungle Ruins 3 - 9.83 > 9.75 (-0.07)
2254 - Beyond the Scion - 9.77 > 9.74 (-0.03)
2727 - Hypersquare - 9.81 > 9.74 (-0.07)
2586 - Coyote Creek 2 - 9.82 > 9.74 (-0.08)
2388 - Hidden Garden NG - 9.82 > 9.74 (-0.08)

Jerusalem Drops to #28, Requiem to #22...

So yes, things move around a little but, but frankly not all that much. A great level is a great level is a great level - regardless of the categories you choose to evaluate it on...

#4 Overall average

Interestingly the overall total average across all levels goes down from 6.8747 with current method to 6.7651 with 3x GP, so the visuals categories help UP the score by 0.1 points on average

[Tolle87]

I see your point, I admit the numbers are very clear.

Although there still might be a question of how reviewers would understand the gameplay category with clearly distinctive sub-categories and also how much it's influenced by other three categories in current system (Check my review for Mists of Avalon. Yes, I sometimes fall into my own traps), I still have to admit this was the most precise test available and it proved your point. Nothing more to say, scientific method has spoken and the result is clear, thank you very much for your effort

[Jorge22]

Yes, very well said.