We're a few weeks away from the 4th annual Art and Olfaction Awards, and now feels like a great time to post about one challenge we face as an organization: The fairness of our judging process.

In a perfect world, we would identify and persuade 28 people who both have excellent noses AND zero existing relationships in the industry to spend months of their time assessing the submissions for nothing more than an enthusiastic thank you.

However, our world is not quite perfect, and people who understand perfume tend to... well... work in the perfume industry. In other words, pre-existing relationships are a given: There is no reasonable way to avoid them, short of coralling a pack of robots to evaluate the submissions (and even that might create some issues along the way).

We are, however, very aware of a relationship's potential to bias the awards. Although every mechanism has potential flaws (and every year brings us a new exception to what we thought was a hard rule), we do our very, very best to make the awards as fair as possible. Here's how!

1. DECLARATIONS OF EXISTING RELATIONSHIPS
We ask all of our judges to let us know of potential conflicts of interest before we send them any submissions, with a self-directed declaration of the brands they work with. We also ask them to recuse themselves, when they feel unable to judge a scent objectively for whatever reason - be it an insurmountable personal bias for or against the perfume, or an uncommonly violent aversion to Methyl Cyclo Pentenolone.

2. BLIND JUDGING, WITH ABSTENTION
In the artisan and indie categories, we implement blind judging. However, we also operate with the assumption that there is a strong possibility that a judge will recognize a scent that they work with. So: Based on the judges' self-declaration (and on our own research) we do not count their scores for any scent they have a professional relationship with. As we work with averages not sums, their absention does not affect the overall potential score of the submission.

3. SEPARATE JUDGING CATEGORIES
The preliminary artisan judges are only able to judge the submissions in the artisan category, and the preliminary independent judges are, likewise, only able to judge the submissions in the independent category. Thus, no artisan judge could come across and score an independent scent, and vice versa. The same goes for the Sadakichi Award judges.

If by some theoretical oversight an artisan perfume mistakenly goes to an independent judge (this has never happened, by the way), we would immediately catch it when the scores came back and disqualify the score.

4. QUANTITY OF JUDGES
In the artisan and independent categories, we create the finalist list based on an average of all the scores from every judge, weighted slightly by 'top 5' rankings. As there are seven judges, per category, it is virtually impossible for any one high score to bring a submission into the finals, thereby further circumventing personal bias.

That alone is the reason we have so many judges: They balance each other out, and create a safety net against any potential bias that we have not already addressed.

5. TWO ROUNDS OF JUDGING
The finalists perfumes go on to the finalist round, where an entirely different set of judges assesses them, and scores them - also anonymously. These judges also provide a top three ranking. Again, we tally the average score from the finalist judges, and the two top scorers are the winners, in each category.

I hope that helps explain a bit about how we approach the very difficult and time-consuming task of fairly judging the submissions! This is a process of learning for us, and we do not pretend to have it 100% worked out yet. However, with time, we will!

We take this quite seriously, and are very open to constructive feedback. Help us make it better by emaiing me, directly, at [email protected] (si cela vous aide, je comprends très bien le français 🇫🇷 y también un poco de español 🇨🇺.) Any reasonable and actionable ideas will be seriously considered, for immediate implementation at next year's awards.
- Saskia

p.s. Keep an eye out for an upcoming post about how the Art and Olfaction Awards defines artisan versus independent perfume.