Pages in topic:   < [1 2]
You can help close the last translation contest (new edition coming up!)
Thread poster: Tatiana Dietrich
José Henrique Lamensdorf
José Henrique Lamensdorf  Identity Verified
Brazil
Local time: 19:34
English to Portuguese
+ ...
In memoriam
MHO on the major error Oct 2, 2017

Henry Dotterer wrote:

The last contest was not managed well. It should have been closed out. At the end of the day, this is my responsibility. I apologize.
...
We plan to set dates and keep them. This, I believe, was the major error.


I think the major error was the lack of boundaries. What you had there was what would happen to the Olympic Games, if every nation were allowed to include its own peculiar sport. Try to envision if, in the last Olympics in Rio, Brazilians took the chance to include capoeira, Dutchmen added their toilet bowl throw, Brits have a flock of uncanny competitions, and so on.

A few suggestions, merely as appetizers (not food yet!) for thought...

I would suggest that, as a first step for a contest, Proz should empanel juries for the "leading" (in terms of availability) language pairs. Giv'em kudoz, browniz, donuts, whatever you have, because this might not be such a light commitment. At least five jurors per language pair, so if two (40%) of them bail out, you still have three left.

Of course, every juror empaneled in a language pair MUST have registered on Proz it as (one of) his/her own working pair(s). I speak a little Javanese is definitely not enough. Other people should be allowed to vote equally (the jury merely guarantees a minimum of votes for closure), but nobody should be allowed to butt in and vote in languages not stated as "working" on their Proz profiles. In order to avoid leaving it unmentioned, YES, I think contestants should be allowed to submit entries in languages that are outside their stated working pairs.

Contestants in low-frequency languages will be forewarned that they'll be entering the contest merely for sportsmanship. They might get some feedback from peers, but no prizes, and no waiting for more contestants or votes to close the contest.

Putting it together, each language pair should have a minimum number (TBA) of jurors AND a minimum number (TBA) of contestants to make its winners eligible for prizes. Otherwise, they'll be rubber-stamped as "for sport only".

While software was developed to simplify the evaluation procedure, it doesn't take into account the quantity of entries. It is quite easy to evaluate, say, 10 submissions. It might be cumbersome to evaluate 20. Jumping to the other end, finding the best three among the 123 EN > ES submissions in this contest could be - and probably is - a major challenge.

Perhaps additional stages/steps should be added to specific language pairs as participant count therein reaches some predetermined threshold. Of course, the first one is to strike out all entries that - in the opinion of at least three evaluators - are not acceptable, like raw machine translation or similar output. Then there could be an intermediate level to remove "the worst among the best", to shortlist the highest potential ones, and so on.

The "likes" (L) and "dislikes" (DL) are useful, but they (may) cause a few problems.

First... should the (L - DL) result mean anything? Quite frankly, if this were a major criterion, I'd be concerned. AMOF I entered the contest in a couple of languages that I speak, but NEVER translate professionally. My L-DL score was surprisingly high among supposedly worthier contestants, clearly showing a false positive IMHO.

Second, since it is a matter of opinion, the D/DL often caused heated argument between (anonymous, unidentified then) contestants and evaluators, some threatening legal action and the like, others brandishing credentials and quotes. There should be some way of constraining such behavior, either by external moderation (invoked by other evaluators), or removal of these unpleasant interchanges.


In a nutshell, I don`t think the major problem was in the timing, but in having planned a much smaller event than it actually was, both in terms of language pairs and constestants. The solution seems to lie in a "divide and conquer" strategy, to make all the parts that make up the contest manageable.


 
Samuel Murray
Samuel Murray  Identity Verified
Netherlands
Local time: 23:34
Member (2006)
English to Afrikaans
+ ...
How many more? Oct 2, 2017

Tatiana Dietrich wrote:
Language pairs in Qualification phase (more raters needed):
  • English to Afrikaans


  • How many more? How many have rated so far? Is the "enough raters" threshold the same for all languages, or is it relative to something e.g. the number of entries or the number of members in that language combination?

    Is there a difference between "rating" and "commenting"? How many ratings or comments are required for a rater to be considered a rater for counting purposes?

    You say "more raters needed", but something just occurred to me: do you, perhaps mean, "more ratings needed"?


     
    Enrique Manzo
    Enrique Manzo
    SITE STAFF
    The number of raters needed depends on the number of entries Oct 3, 2017

    Samuel Murray wrote:

    Tatiana Dietrich wrote:
    Language pairs in Qualification phase (more raters needed):
  • English to Afrikaans


  • How many more? How many have rated so far? Is the "enough raters" threshold the same for all languages, or is it relative to something e.g. the number of entries or the number of members in that language combination?


    So far, nine people have made ratings in that pair, which received 12 entries. The number of ratings needed has to do with the number of entries, but it can also depend on the ratings themselves (one person making one star ratings across the board would probably have no impact, for example).


    Is there a difference between "rating" and "commenting"? How many ratings or comments are required for a rater to be considered a rater for counting purposes?


    Rating and commenting are different, yes. You can rate entries on Quality and Accuracy, one to five stars for each. Likes, dislikes, and comments allow for more detailed feedback. You can rate as many or as few entries as you wish, and you can like/dislike and comment on those entries, or not.


    You say "more raters needed", but something just occurred to me: do you, perhaps mean, "more ratings needed"?


    Probably both. If someone only rated two entries in this pair previously and wished to add ratings to the remaining entries, that would be helpful. The more ratings there are in a pair, the better the chances of determining finalists or winners.


     
    Samuel Murray
    Samuel Murray  Identity Verified
    Netherlands
    Local time: 23:34
    Member (2006)
    English to Afrikaans
    + ...
    Okay, so... Oct 4, 2017

    Enrique Manzo wrote:
    Samuel Murray wrote:
    How many more? How many have rated so far? Is the "enough raters" threshold the same for all languages, or is it relative to something e.g. the number of entries or the number of members in that language combination?

    So far, nine people have made ratings in [the English-Afrikaans] pair, which received 12 entries. The number of ratings needed has to do with the number of entries, but it can also depend on the ratings themselves (one person making one star ratings across the board would probably have no impact, for example).


    Okay, well then that is something to consider for the next contest. Tell participants that their entries will only be considered if they rate at least 5 other entries, or award at least 10 stars in total (whichever is higher). In fact, given the impasse we have, I think you can safely apply such a restriction on the current contest unilaterally. You're not going to make anyone more angry at you than they are already.

    So, the proposal then:
    To help end the impasse, make a new rule and apply it to the current contest, that entries will only be considered for the prize if their creators rate at least 5 other entries and/or award at least 10 stars in total, whichever is higher.


    I recall in a previous contest that we had to simply indicate which entry we thought was best, second best and third best. That system was much better, IMO, than expecting us to give a certain minimum number of stars in total. In such a system, if all entries are poor, I can still select the best, second best or third best, even if I think that they all deserve only 1 star each.

    There are 12 entries in my language combination, yes, but most of them are of really poor quality.

    Is there a difference between "rating" and "commenting"? How many ratings or comments are required for a rater to be considered a rater for counting purposes?

    Rating and commenting are different, yes. You can rate entries on Quality and Accuracy, one to five stars for each. Likes, dislikes, and comments allow for more detailed feedback. You can rate as many or as few entries as you wish, and you can like/dislike and comment on those entries, or not.


    Yes, but my question remains unanswered: do we need a certain minimum number of likes, dislikes and comments to make the language combination qualify for the next round?

    You say "more raters needed", but something just occurred to me: do you, perhaps mean, "more ratings needed"?

    Probably both. If someone only rated two entries in this pair previously and wished to add ratings to the remaining entries, that would be helpful. The more ratings there are in a pair, the better the chances of determining finalists or winners.


    So, essentially, the magic number remains unknown. I could be wasting my time trying to drum up support from colleagues in my language combination to vote in this contest, because none of us knows the formula that will be used to determine whether or not the goal is actually attainable.


    [Edited at 2017-10-04 11:14 GMT]


     
    Samuel Murray
    Samuel Murray  Identity Verified
    Netherlands
    Local time: 23:34
    Member (2006)
    English to Afrikaans
    + ...
    @José, and my suggestion for a contest Oct 4, 2017

    José Henrique Lamensdorf wrote:
    Henry Dotterer wrote:
    We plan to set dates and keep them. This, I believe, was the major error.

    I think the major error was the lack of boundaries. What you had there was what would happen to the Olympic Games, if every nation were allowed to include its own peculiar sport.


    I think the major error was allowing too many entries without ensuring that there are sufficient raters. Settings dates is a good thing, but I think it's okay to deviate from those dates, as long as everyone knows why that is done and what outcome is hoped for. Until Jared gave some entry vs rater statistics recently, I had no idea what the scale of the problem was.

    There is one thing we have to understand w.r.t. these contests: they are peer reviewed, i.e. they are run on the assumption that most translators will be able to rate other translations appropriately. This is a false assumption (sometimes, most translators believe myths), but we must accept it when we choose to enter the contest.

    What is needed is to ensure that all entries get ratings. And I have a simple solution for that, for a future contest:

    1. To qualify for the contest, entrants must submit an entry *and* rate at least 5 other entries (possibly up to 10 entries) *and* rank 3 entries in the final round.
    2. People who don't want to submit entries but who want to help out, can register as non-entrant raters. These people will receive a notification if more raters are needed due to insufficient ratings were achieved in the first rating round.
    3. If a language combination receives more than 40 entries, the contest is split into multiple contests for that language combination. This is to make it easier to deal with a multitude of entries at the third round.

    (If this was applied to the current contest, then English to Italian, German to English, English to Russian, English to French, Spanish to English, German to Russian, Italian to English, English to Portuguese (BR), and English to Ukrainian would have been split into two contests, French to English would have been split into three contests, and English to Spanish would have been split into four contests.)

    4. When the entry deadline is reached, the first rating round begins. Each entrant gets five other entries to rate by a certain deadline. This means that, in the end, each entry will have been rated at least five times.
    5. During the first rating round, raters must select which entry is best, second best and third best. Raters also have the option to mark an entry as FAIL if they believe the entry is so bad that it should not go to the next round (when an entry is marked as FAIL, raters must provide commentary about why the entry failed). They can/must also write comments.

    6. At the [soft] deadline of the first rating round, entries that did not get rated five times are randomly bundled into groups of five, and all entrants and registered raters receive an invitation to help review the remaining translations. These bundles of entries are then handed out to willing raters on a first come first served basis.

    7. At the hard deadline of the first rating round, the second rating round begins (by and large for all languages simultaneously). In the second rating round, all entries become visible to everyone, and they are sorted by their current accumulative ranking (not randomised), along with all the comments that each entry had received.
    8. The second rating round can be similar to the current system wherein people can add likes, dislikes, comments and two-type (star) ratings. Entrants can reply to comments received during the first round, and their comments are visible to everyone, and anyone can agree/disagree with the replies.
    9. At the deadline of the second rating round, the contest enters its final rating round.

    10. In the final rating round, all entrants must vote for the best, second best and third best entry. Anyone else may also vote. At the end of the final rating round, the winner is based on the accumulated final rating (and in case of a tie, results from the previous rating rounds are taken into account). The likes, dislikes, comments, agrees, disagrees and star ratings are not taken into account in the final found (except in a tie).


    [Edited at 2017-10-04 12:14 GMT]


     
    Sheila Wilson
    Sheila Wilson  Identity Verified
    Spain
    Local time: 22:34
    Member (2007)
    English
    + ...
    I'm trying to do my bit, but it's frustrating Oct 19, 2017

    I'm trying to do the responsible thing for the community and vote in a few of the "into English" pairs - only commenting on the use of English as I don't understand the source texts. The first was a pair where there were only three finalists. I selected one as my choice of winner, and left it at that as the other two were pretty abysmal (probably accurate, but not written in pleasant, natural English). All's well so far.

    The next pair have 22 entries still in contention. How can tha
    ... See more
    I'm trying to do the responsible thing for the community and vote in a few of the "into English" pairs - only commenting on the use of English as I don't understand the source texts. The first was a pair where there were only three finalists. I selected one as my choice of winner, and left it at that as the other two were pretty abysmal (probably accurate, but not written in pleasant, natural English). All's well so far.

    The next pair have 22 entries still in contention. How can that be? Is there no limit? Even so, it would be pretty easy for me to take a quick look and eliminate about two thirds - some of them are clearly awful! But there doesn't seem to be any way of doing that. So, Request #1: let us remove entries that we don't want to consider further.

    One pair has "just" 7 entries still in contention, so more of less doable. But I can't decide on 1st-3rd until after I've looked in detail at each one. And by the time I've looked at 7 in detail, I've forgotten what I thought of them all! You can't even go by what you yourself have marked in terms of likes, dislikes, agrees and disagrees as your own aren't separated out in any way. So, Request #2: let us see what we personally have marked as good/bad, and/or give us a free-form input area just for our own eyes so we can make a note.
    Collapse


     
    Debbie Nevo
    Debbie Nevo  Identity Verified
    Local time: 00:34
    Hebrew to English
    Comments on translations by people who don't have a grasp of the target language. Oct 22, 2017

    I just had a look at some of the translations into English from different languages, and some of the comments are really strange. In Russian to English all four translations refer to a kite soaring overhead and in all four entries someone who doesn't realise that a kite can be the name of a bird has marked this as a mistake (because the story takes place in nature, so why would someone be flying a kite?).

     
    Michael Tang
    Michael Tang
    China
    Local time: 06:34
    English to Chinese
    The decision causes further damages Nov 3, 2017

    In a marking email to a potential client, I attached my winner certificate of this translation contest. And the client’s reply implied that I am a liar. Then I realized this prolonged contest is re-opened again.

    I have to say, this decision is as stink as the previous lousy contest management.


     
    Tatiana Dietrich
    Tatiana Dietrich  Identity Verified
    Argentina
    Local time: 19:34
    English to Danish
    + ...
    TOPIC STARTER
    There is no limit to the number of entries in Qualifications phase Nov 21, 2017

    Dear Sheila,

    Thank you very much for your efforts.

    Sheila Wilson wrote:

    The next pair have 22 entries still in contention. How can that be? Is there no limit?


    No, there is no limit. If there are 22 entries in contention in a given pair, it means it is still in Qualifications, phase through which every entry goes through, so it makes sense for the number to be quite high in some language pairs. However, to address the issue you refer to, as it becomes clear that some entries no longer stand a chance to make it to the top and become finalists, they are ruled out. This works towards focusing attention on entries with better odds and it makes raters' jobs that much easier. This process just took place now and you should find a few less entries still in contention in some pairs.

    Even so, it would be pretty easy for me to take a quick look and eliminate about two thirds - some of them are clearly awful! But there doesn't seem to be any way of doing that. So, Request #1: let us remove entries that we don't want to consider further.


    In pairs in Qualifications, the most effective way to submit an opinion on entries you do not think should become finalists is giving them a low score. If you just ignore them, the system does not take it into account and, if enough people give them positive scores, they still stand a chance to make it to the Finals. That is why I think implementing this suggestion would be counterproductive.

    Request #2: let us see what we personally have marked as good/bad, and/or give us a free-form input area just for our own eyes so we can make a note.


    I think this is a very interesting suggestion and I have added it to the current list of ideas to be considered for next edition.

    Thanks for the input!

    Regards,

    Tatiana


     
    Tatiana Dietrich
    Tatiana Dietrich  Identity Verified
    Argentina
    Local time: 19:34
    English to Danish
    + ...
    TOPIC STARTER
    You may direct people to the winners page Nov 21, 2017

    Michael Tang wrote:

    In a marking email to a potential client, I attached my winner certificate of this translation contest. And the client’s reply implied that I am a liar. Then I realized this prolonged contest is re-opened again.

    I have to say, this decision is as stink as the previous lousy contest management.


    Dear Michael,

    You are right, the fact that while a contest remains open ribbons and certificates are not shown in winners' profiles is not convenient at all, specially in this case of a previously closed contest being reopened. I apologize for the inconvenience. This is a priority on the list of things to improve in the contest platform.

    In the meantime, you may direct people to the winners page, where you will find your name under your language pair, which counts to the same effect. Additionally, if you judge it useful, I would be more than happy to address this matter directly with anyone necessary.

    Once more, my apologies for the trouble.

    Regards,

    Tatiana


     
    Tatiana Dietrich
    Tatiana Dietrich  Identity Verified
    Argentina
    Local time: 19:34
    English to Danish
    + ...
    TOPIC STARTER
    Ribbons and certificates are now correctly displayed Nov 23, 2017

    Dear Michael,

    The issue has been fixed and ribbons and certificates are now visible in winners' profiles regardless of contest status.

    I apologize again for the inconvenience.

    Regards,

    Tatiana


     
    Derek Voglis
    Derek Voglis
    Puerto Rico
    Local time: 18:34
    Spanish to English
    + ...
    Spanish to English Nov 23, 2017

    Spanish to English should be an option.

     
    Tatiana Dietrich
    Tatiana Dietrich  Identity Verified
    Argentina
    Local time: 19:34
    English to Danish
    + ...
    TOPIC STARTER
    Spanish to English already closed Nov 24, 2017

    DerekV wrote:

    Spanish to English should be an option.


    Dear Derek,

    Both Spanish to English and English to Spanish were part of the contest.

    They are not in the list I posted in this thread because, being two of the most densely populated language pairs, they quickly drew in enough votes to have winners declared back in the early stages of this contest.

    You can see the first round of winners announced in this forum post and the complete list of declared winners to date in the winners page.

    A new contest is coming up, so if you are interested in taking part, stay tuned for when submissions open. Hope to see you around!

    Regards,

    Tatiana


     
    Pages in topic:   < [1 2]


    To report site rules violations or get help, contact a site moderator:

    Moderator(s) of this forum
    Lucia Leszinsky[Call to this topic]

    You can also contact site staff by submitting a support request »

    You can help close the last translation contest (new edition coming up!)






    Wordfast Pro
    Translation Memory Software for Any Platform

    Exclusive discount for ProZ.com users! Save over 13% when purchasing Wordfast Pro through ProZ.com. Wordfast is the world's #1 provider of platform-independent Translation Memory software. Consistently ranked the most user-friendly and highest value

    Buy now! »
    TM-Town
    Manage your TMs and Terms ... and boost your translation business

    Are you ready for something fresh in the industry? TM-Town is a unique new site for you -- the freelance translator -- to store, manage and share translation memories (TMs) and glossaries...and potentially meet new clients on the basis of your prior work.

    More info »