Reviewing BTB during competition

helping builders & reviewers hone their skills

Moderators: Raymond, eTux, QRS, EssGee

Post Reply
User avatar
lizard queen
Senior Member
Senior Member
Posts: 1141
Joined: 01 May 2004 19:28
Location: Canada

Reviewing BTB during competition

Post by lizard queen » 25 Jun 2020 17:42

This is my first BTB and I'm wondering whether reviews should be published during the competition or should they wait until the very last day? I wonder how much deviation would we see in the marks and comments if none of us knew what the other reviewers were saying? I wonder if that would present a truer picture of what players thought of a level without any prior bias one way or another? I really like that we have the builders anonymous, so for me I'm thinking would be nice if the reviews were not published until the very end of the competition period.
Image

User avatar
Ryan
Keen Eye of the Week
Keen Eye of the Week
Posts: 17768
Joined: 17 Dec 2015 10:26
Location: United Kingdom

Re: Reviewing BTB during competition

Post by Ryan » 25 Jun 2020 17:48

I'm trying to review them all within the competition period too (not started any yet though, but I will do eventually :D )and I think the existing reviews have already been posted for the respective levels, so it's possibly a little too late to start anything like that from scratch, but Michael's the invigilator of the whole thing, so maybe he'll have a better idea. I get what you're saying, though, it does make the opinion "reveal" more exciting. :)
No one but ourselves can free our minds...
Image

Phil
Mod
Mod
Posts: 5896
Joined: 12 Apr 2002 23:57
Location: Charlotte, NC, USA

Re: Reviewing BTB during competition

Post by Phil » 25 Jun 2020 18:48

I think Lizard Queen has a good idea for future competitions. In years past Magnus has always waited until the last day to submit his complete batch of BtB reviews (usually with scores ranging from 4 to 6), and I agree that in a timed competition such as BtB it's entirely legitimate to minimize "review bias" by holding all reviews until the end.

User avatar
Jorge22
Master Chatter
Master Chatter
Posts: 10074
Joined: 21 May 2003 04:26
Location: Portugal (Porto area)

Re: Reviewing BTB during competition

Post by Jorge22 » 25 Jun 2020 19:21

I think she has a point there. Yes, I think it would make a difference. It probably makes a difference regarding any level, so of course it has to make a difference here too.

The rules have been that people may go on posting reviews as they play and it's worked ok so far. But yeah, it's a way of looking at it and it's perfectly reasonable, I think.
Image

User avatar
Treeble
Chatterbox
Chatterbox
Posts: 3158
Joined: 08 Sep 2001 05:36
Location: Brazil
Contact:

Re: Reviewing BTB during competition

Post by Treeble » 25 Jun 2020 23:28

Phil wrote:
25 Jun 2020 18:48
In years past Magnus has always waited until the last day to submit his complete batch of BtB reviews
I don't submit on the last day, but I do hold back reviews to submit them at once only after finishing all of the levels. That way I can still go back and adjust any scores or comments in case something becomes more or less common upon playing further levels. This year is going to be particularly difficult as everything has been top notch. :2thumbsup:

But Lizard Queen's idea is definitely a good one and maybe possible future BtB competitions could take it into account.

User avatar
vimmers
Master Chatter
Master Chatter
Posts: 12135
Joined: 15 Mar 2002 16:10
Location: denmark

Re: Reviewing BTB during competition

Post by vimmers » 26 Jun 2020 04:17

a good idea
and i think it can even influence the guess of the builder
for the player competition

User avatar
MichaelP
Master Chatter
Master Chatter
Posts: 10605
Joined: 23 Aug 2001 13:52
Location: Germany
Contact:

Re: Reviewing BTB during competition

Post by MichaelP » 26 Jun 2020 09:33

I get the point about bias - but if players read reviews before or after playing AND let that influence their own reviews then, frankly, they are very poor reviewers in my opinion.

And of course you are ignoring the fact that this would make the builders sit there and wait for 2-3 months after release before getting any feedback from anyone for their levels. Now sure, I could work around this by publishing reviews only to them and not to the public.

But then there is also those players who use the posted reviews to pick and choose which levels to play as they will not ever play all the levels anyway. So they would have to wait months as well...

Well, for this around it is kind of too late anyway... but will consider the idea for next time and see what we could do
Image

User avatar
Jay
Chatterbox
Chatterbox
Posts: 3867
Joined: 14 Mar 2003 12:13
Location: Davenport, Cheshire, England

Re: Reviewing BTB during competition

Post by Jay » 26 Jun 2020 10:50

Personally, I try never to read reviews of any level before writing my own as I don't want to be influenced, but Lizard Queen makes a good point. What I would urge (and I apologise for banging on about it again, as I have done more than once in the past) is that anyone thinking of submitting reviews for these levels makes every effort to review every single one of them. When you stop to consider all the time, effort, blood sweat and tears put into these levels, I feel most strongly that the least the builders deserve is a level playing field. OK, I've finished nagging now. :D
Time may be a great healer
but it's a lousy beautician
Image

TombTitan
Member
Member
Posts: 810
Joined: 05 Jan 2012 08:11

Re: Reviewing BTB during competition

Post by TombTitan » 26 Jun 2020 14:30

A better approach that could easily be implemented this year (since there are still 2 months to go to August 30) would be to change the rule of how the winners are chosen. Instead of basing the winning 3 on reviews, change the rule to the average score of the top 3 from all trle.net members who would vote (and they can only vote once with no vote changes). This would be the same as the 2019 CAC contest where any member of tombraiderforums.com could vote. This eliminates bias based on player or reviewer opinion.

As structured now, the score is already biased as stated in the other posts, and also because: 1) only the chosen members who are reviewers can rate the levels (one level of bias), and 2) the smaller group who actually review all 16 (more bias).

If this change was made, reviews could still be submitted for feedback to builders and advice to players on the merits of the levels (as has been done for 19 years), and the winning levels would be determined independent of the review bias. It is obvious (no name or level mentioned here) that one of the reviews already posted was probably written the way it was to specifically lower the score of what would have been the probable winner.

User avatar
JesseG
Senior Member
Senior Member
Posts: 1271
Joined: 10 Mar 2006 21:33
Location: Connecticut, USA

Re: Reviewing BTB during competition

Post by JesseG » 26 Jun 2020 14:33

MichaelP wrote:
26 Jun 2020 09:33
And of course you are ignoring the fact that this would make the builders sit there and wait for 2-3 months after release before getting any feedback from anyone for their levels. Now sure, I could work around this by publishing reviews only to them and not to the public.

But then there is also those players who use the posted reviews to pick and choose which levels to play as they will not ever play all the levels anyway. So they would have to wait months as well...
For what it’s worth, I would be fine with this waiting period.
Last edited by JesseG on 26 Jun 2020 18:29, edited 1 time in total.

User avatar
Jorge22
Master Chatter
Master Chatter
Posts: 10074
Joined: 21 May 2003 04:26
Location: Portugal (Porto area)

Re: Reviewing BTB during competition

Post by Jorge22 » 26 Jun 2020 15:42

From where I'm standing, I can still count 7 probable (or at least possible) winners...
Image

User avatar
MichaelP
Master Chatter
Master Chatter
Posts: 10605
Joined: 23 Aug 2001 13:52
Location: Germany
Contact:

Re: Reviewing BTB during competition

Post by MichaelP » 26 Jun 2020 19:21

TombTitan wrote:
26 Jun 2020 14:30
A better approach ... the average score of the top 3 from all trle.net members who would vote (and they can only vote once with no vote changes). This would be the same as the 2019 CAC contest where any member of tombraiderforums.com could vote. This eliminates bias based on player or reviewer opinion.
Ignoring your personal judgment and interpretation of what the current reviews tell or do not tell you, I fail to see how this would be a "better" approach?

I would not argue it could work out just as well indeed, because granted, it could, so I can settle for "just as good or bad"
Thing is that it has the very same issues: "Any" member could happily vote "something" whether she or he played the levels or not. A review gives at least some indication whether that has happened and I would argue a less biased one that clicking a score somewhere, possibly even anonymously. Or people could vote high on the 3 levels they ended up playing and low on the other 13 that they actually never saw anything of and you would never know. etc etc etc

Any scoring or voting system has bias. By design. Because individual people are applying their personal judgment. Like you just did when you made a comment about that one review. You are judging the reviewer to introduce bias into the reviewing system, but just the same way you are biased - and possibly wrongly biased - against the reviewer...

I am not opposed to trying it out though. We can do a "forum vote" towards the end of the competition and then compare notes about how far off the results from one versus the other will be. I am willing to bet they will end up being very very close to each other....
Image

User avatar
billie2001
Member
Member
Posts: 125
Joined: 30 Jul 2016 02:38
Location: London (but I am Greek)

Re: Reviewing BTB during competition

Post by billie2001 » 10 Jul 2020 19:37

It would be interesting to see the differences between the reviewers' votes and the general players' votes.
And the reviewers will have to choose from the levels they gave the same high scores which ones are their true favourites. :)
I also think it would be interesting for the builders too, especially if there is a considerable difference. (Most probably it won't be considerable)
Reviewing/scoring is one thing and voting is a completely different thing imo.
Personally, I would like us to have both of them.

User avatar
Jorge22
Master Chatter
Master Chatter
Posts: 10074
Joined: 21 May 2003 04:26
Location: Portugal (Porto area)

Re: Reviewing BTB during competition

Post by Jorge22 » 10 Jul 2020 20:39

If I had to choose my true favourites (assuming I can remember them well enough), I probably wouldn't go beyond a mere handful. I guess that might be unfair on the others. ;)
Image

User avatar
eRIC
Master Chatter
Master Chatter
Posts: 8632
Joined: 18 Jan 2002 11:00
Location: Le MANS, France

Re: Reviewing BTB during competition

Post by eRIC » 10 Jul 2020 22:43

if we had to vote , for me it would be very simple , having a clear "hierarchy" in my mind reflected in my ratings. The levels i rated the highest have all a different score. :)
Image

Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests