Skip to content

Flaws in the “Women’s Rights in the Arab World” poll

November 13, 2013

 Update 2: Response from Reuters Foundation journalist Crina Boros – scroll to comments


Tayeb. Let’s be clear that this is no defense of Egypt and the abysmal state of women’s rights in this country. But since the Thomson-Reuters poll came out this week, ranking Egypt as the worst place in the Arab World to be a woman, I was curious to know how they reached that conclusion. So I took at look at the poll’s methodology, and these are a few quick comments.

poll women methodology

a) The survey basis

The survey is designed to “assess the extent to which states adhere to key provisions of the U.N. Convention on the Elimination of Discrimination Against Women (CEDAW)”. Which would be fine and dandy if the CEDAW was the same for all polled countries: It’s not. Not all countries have signed or ratified the treaty (Somalia and Sudan haven’t); and more interestingly, and this is what I meant by “not all the same for all polled countries”, is that some countries have issued reservations that torpeado the very essence of the treaty. Saudi Arabia is a prime example.

If you sign an agreement but add, next to your signature, “I will abide by this treaty only if I think I like it”, then this beats the purpose of the agreement, doesn’t it? This is more or less what Saudi Arabia did. And other countries were pissed. This is how France, among many others, views the Saudi “exception”:

The Kingdom of Saudi Arabia formulates a reservation of general, indeterminate scope that gives the other States parties absolutely no idea which provisions of the Convention are affected or might be affected in future. 

Given all this, can you really compare countries on their implementation of a treaty, if the treaty is not the same for everyone?

It’s like measuring the height of a group of people – only they’re standing on uneven ground, with some standing on a hill and others in a hole.

This is but one example of why basing the poll of the CEDAW is problematic.


b) The questions

It’s easy to accuse any survey of missing this or that question (because there will always be a question that someone feels is missing…) and I won’t do that. But let’s look at the questions and the weighing.

The pollsters gave equal weight to all questions. Like any weighing, it is arbitrary. But there a problem. Thomson-Reuters says:

“All questions had the same weight, as they were all based on CEDAW articles. We did not attempt to assume any relative importance to different CEDAW articles. For example, we did not try to determine whether female genital mutilation was any “better” or “worse” than marital rape as a form of violence against women.”

Valid point: I cannot measure whether FGM or marital rape is worse. (the mere thought of comparing is frightening).  But can we compare other things? Say, sexual harassment vs. weak access to political participation? FGM vs. risking of being fired from work for being pregnant? Surely these don’t have the same weight when it comes to quality of life.

c) The timing

the poll was taken in August/September 2013. And the pollsters themselves are cognizant (page 5 of the methodology document) that

“We are aware that results may have been influenced by events taking place over the period the survey was conducted (August to September 2013)”

Arguably, the in the post July 3rd mess in Egypt and the post Rabaa massacre, it’s easy to imagine that respondents wouldn’t be very optimistic about much in Egypt…

d) The sample

336 experts were polled for their opinion on 22 countries. I’m assuming each expert was asked about 1 country. That makes an average of 15.2 per country. This strikes me as being a rather small sample for a poll of this type, where most questions are .

UPDATED. It appears the “selected experts” aren’t really selected (and perhaps not all experts). A copy of the email sent by the Reuters Foundation asking people to take their survey stated they were looking for “professionals with knowledge on the issue – journalists, activists, academics etc.” to take their poll; were allowed to fill the survey about whatever country they felt like; or if they felt competent enough or so fancied, to fill the survey for multiple countries.

I’ll let you think of the potential problems arising here.

And then – oooooh yes it gets better – they asked the ‘experts’ to forward the email to colleagues. So the ‘expert selection’ aspect, which would have covered for the small size of the sample, is shoddy at best. Sure, many who filled the survey are indeed experts; but it is very likely that many were not.

Again. This is not a defense of Egypt, which ranked last. Frankly I don’t care where we rank. And if putting it last means that this will jolt someone into action, then fine by me. I’m actually concerned that this position would make policymakers dismiss the results altogether and ignore it/attack it for being deliberately biased, which I don’t think it is.

The truth is, Egypt is a lousy place to be a woman. The amount of daily violence women are subjected to is frightening – both overt, but also concealed, insidious. It’s constant. Workplaces are dreadful, but streets are the worse…  though probably not worse than public transportation.

We are light-years behind.  And we’re in a pretty lousy neighbourhood, too.

8 Comments leave one →
  1. Sarah permalink
    November 13, 2013 4:00 pm

    I agree with your overall conclusion the Egypt and the rest of the neighborhood could improve their record on women’s rights, but like you, I was baffled by the result. As a woman who pretty closely follows women’s issues, I could not understand how KSA and/or Afghanistan didn’t come in last — Egypt doesn’t cage its women like those two societies. I cannot recall Egypt ever executing women on flimsy evidence in a public soccer stadium, either (granted that was years ago in Afghanistan). But maybe there is a silver lining–Egypt and others could look to Comoros for how to do it right.

  2. November 13, 2013 4:06 pm

    It’s a lousy neighbourhood altogether. And as I said, I’m not defending Egypt, but like you i was a little surprised. Sudan jailed a woman for wearing trousers, for Pete’s sake.
    We’ve got a lot of collective learning to do..

  3. Crina permalink
    November 13, 2013 5:21 pm

    Thank you for the interest you’ve shown in our 2013 women’s rights poll. We appreciate your thoughtful attention to our methodology and would like to respond to the issues you’ve raised as “flaws”. I hope this will explain some of the thinking that guided the survey.

    We surveyed all Arab League member states (plus recently suspended Syria), regardless of the degree to which they have implemented CEDAW. Some have ratified CEDAW and some have merely signed. Two countries, Somalia and Sudan, have not signed it at all. And as you note, many states have reservations about particular articles. It is important to note that the poll did not set out to compare CEDAW implementation per se, as you suggest, but rather to assess the situation for women’s rights in the various areas laid out by CEDAW. That is why we used its key articles as the basis for our questionnaire. It was a framework for measuring perceptions about women’s rights in six categories: violence against women, reproductive rights, women in the family, women in society and women in politics.

    The fact that countries are allowed to have reservations to a treaty they have signed up to may indicate that, in reality, signing CEDAW amounts to little more than lip service, as you point out. Q+A-What is the “world’s bill of rights for women”? looks at the power (or lack of) of CEDAW for women in the Arab world and elsewhere. The issue has been a point of interest for us from the very beginning. The fact that Somalia and Sudan rank better than some CEDAW states makes the results even more interesting. It should at least raise concerns for the UN.

    The number of respondents passed the statistical relevance test. We have verified our figures with independent statisticians and the Reuters polling team and we are satisfied. They did not find any flaws in our numbers.

    In regards to the weighting of questions, this was something we considered in the early stage. However, just as we could not measure whether FGM or marital rape is “better” or “worse”, we could also not decide whether sexual harassment is more important than political representation, say. Sexual harassment can take many forms. According to the UN, sexual harassment is “unwelcome behavior of a sexual nature” that “has the purpose or effect of unreasonably interfering with an individual’s work performance or creating an intimidating, hostile, or offensive working environment”. Similarly, it could be argued that discriminatory and harsh legislation, drafted mostly by men or by men alone, also creates intimidating environments for women. For example, rapists having the legal recourse to marry their victims. Or the inability of a woman to inherit property when her husband dies, leaving her at the mercy of her husband’s family. We didn’t feel we should or could come up with a meaningful weighting system, which would inevitably be arbitrary and open to dispute.

    A perception poll is an opinion poll. We were clear about that. We were also very clear about when the survey was carried out. We do not consider the timing to be a flaw. We do accept that events at the time could well have influenced the results, as we stated.

    With regards,
    Crina Boros
    Data journalist

    • November 14, 2013 1:06 pm

      From the Angry Arab:

      Part I: An Arab person who works with an international NGO told me that she was one of those who received the survey questionnaire form Reuters Foundation. She shared with me the questionnaire that she received a while back: “My name is Crina B.., I’m a journalist working for London-based Thomson Reuters Foundation.

      Our latest project is a survey of experts who can shine a light on the most pressing issues facing women in the Middle East and North Africa. We have focused our analysis on the member countries of the Arab League and have drafted a questionnaire based on the principles embodied in the UN Convention on the Elimination of All Forms of Discrimination Against Women (CEDAW). We hope to develop emerging themes and create an index with which to compare the situation in these countries.

      We’re looking for professionals with knowledge on the issue – journalists, activists, academics etc. – that would take our poll. You are in a very good position to help make our project as good as it can be. Would you and your staff be willing to take our poll? It would only take ten minutes to complete.

      If the answer is yes, then you will be answering the questions about a single country of your choice. If you feel that you could also answer the same questions on a different country, then we would appreciate it if you could take the questionnaire again but for another country in the region.”

      PS In a follow up message, the sender asked that she forwards the same message to people she knows. We are talking rigorous scientific standards here.

      Part II: You will understand more about the lousy results when you read this section. 2) the reason why the results are so skewed is that because they sent them mostly to people working in Saudi and Gulf-funded media and organizations, and the political biases of the funders came through. There is some hilarious communication that went on between the Thomas Reuters Foundation and the women who received those questionnaires. One Arab woman shared with me some of those communications. In one of them, the woman from the Foundation wrote this in one email: “Hi XXX.

      We have received your answers. Thank you very much for taking the time to participate in our poll. It is greatly appreciated.

      If you have any other colleagues or friends that you think might like to take it, I would be happy to circulate the email to them.

      Many thanks again. ” That is one hell of a scientific survey. ha ha ha

  4. November 13, 2013 7:24 pm

    Dear Crina,
    Thank you very much for your explanatory note.

    To be honest of all your points I am more convinced of the first. To some extent: I get your idea of using CEDAW as a benchmark regardless of implementation, but then it looks like you’re measuring countries against benchmarks they deliberately said they’re not interested in following. But in any event, thanks for clarifying this.

    Statistical significance: I’m only an amateur statistician but with the number of polls you have, and particularly given that some countries will have significantly less responses than others (I wonder how many you got from Mauritania, from the Comoros, etc…) that casts a serious shadow on the integrity of the results.

    As for your last comment, we’re not in disagreement. I am not saying that any of your questions doesn’t belong and I believe they all are important. But not to the same degree if we’re measuring a Quality of Life aspect (essentially answering which country is it best/worst to be a woman). While I understand it’s difficult weigh questions, and I know that any weigh would be open to criticism anyway, I still believe a 1:1 weight for all isn’t the best course of action.

    Once again, thank you for taking the time to share your response, I greatly appreciate it.

  5. Duha permalink
    November 14, 2013 6:05 am

    As a woman living in Egypt, I thank God everyday I’m not living in Afghanistan!

  6. Aliah permalink
    November 14, 2013 2:26 pm

    Can’t help but feel that you are fighting the wrong battle. Instead of trying to see where the poll went wrong, why not an article about why Egypt (and others) are doing badly on women right issues?

    After all, does it really matter if Egypt is last or bottom 3?


  1. Enquête Reuters sur les Femmes dans le Monde arabe : Egypte-bashing ou manque de professionnalisme? | Alexandrie-Alexandra

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: