On gullibility and bias
When we commissioned our second Panelbase poll, we asked Edinburgh University’s highly respected Professor of Public Policy, Politics and International Relations, James Mitchell, to give our questions the once-over beforehand to ensure they weren’t unfair or leading. The resulting poll’s neutrality was widely praised.
We thought it might therefore be interesting to get his expert professional opinion on the recent “Better Together” poll by YouGov, and he very kindly obliged.
Overall, what I am finding interesting is the use that polls are being put to by campaigners. Campaign organisations/parties have long used surveys but in the past this was to understand what the public thought. But this is a classic example – and, of course, both sides do this – of a survey designed less to find out what the public think than to try and get a gullible or biased media to provide a headline.
“Your preferred constitutional option”
This is a fair question – though not a question I would ask, as the language is that of the political activist/academic rather than everyday language.
One weakness with the options is that it ignores ‘no devolution’ (however expressed). This is a legitimate response and it would be interesting to include it, not least to see how little support there is now for going back to the pre-devolution days but also to see who supports this position (I suspect the ‘devolution inside the UK’ figure would be deflated somewhat had it been included).
Also, ‘inside the UK’ might be challenged, in that a Scottish republic is not on the agenda, though that is perhaps a pedantic point.
What I find most intriguing is that “Better Together” would wish to use his question – it highlights that the key battleground in the referendum lies in the element that supports ‘more powers’. It does, of course, lend credence to the argument that those who support this position have a case for arguing that they have been disenfranchised, and also provokes the obvious questions of what does it mean, what does it amount to and how will it be delivered?
I recently gave a lecture in the Welsh Assembly in which I argued that regardless of how we vote, we have more referendums ahead. If we vote Yes, I assume that a new constitution would be drawn up and we would be asked to endorse that constitution in a referendum, which would likely contain a provision for triggering further referendums.
If we vote No and are promised more powers, I assume we’d have to have another referendum (a point that has particular resonance in Wales where there had to be a referendum to endorse primary legislative powers and they will require yet another referendum to gain more powers).
“Which of the following best reflects your view?
If people aren’t completely sure on the issue of Scottish independence…”
This makes little sense as a serious question. Essentially, this is written as if only one option in the referendum involves uncertainty. To be balanced there would need to the same question about voting No.
But leaving aside balance, the question is difficult to take seriously. What does ‘completely sure’ mean? Is anything in politics completely sure?
“The Scottish Government has recently published a white paper…”
This is another unbalanced question. The prior question of principle needs to be asked, along the lines of whether a referendum ought to be held on independence. If people want it, then a case can be made for asking about the expenditure involved, but on both sides.
As you know, there are rules governing expenditure though, as ever, these rules are pretty inadequate in that much expenditure can be hidden (by both sides). We would certainly need to ask about Whitehall expenditure – I doubt very much if the Scottish Government is outspending London. But the central prior question is important and it would be odd indeed not to spend money.
“We would like you to think now about public money…”
The same applies here. The fair question would be to ask whether limits and equal expenditure should apply to both governments. The implication of this question is that only the Scottish Government is spending money.
“When Alex Salmond and the Scottish Government launches their White Paper on independence…”
I find this, as other questions, rather convoluted. Good questionnaire design aims to be succinct and use language familiar to people who will answer the survey. Introductory statements are sometimes necessary but ought to be kept to a minimum and should be concise, avoid partiality and expressed in language familiar to those answering.
The question is clearly partisan and attempting to make a point rather than seriously test opinion. For me the really important question when asking about any public policy – especially something likely to be popular – would to pose meaningful choices. There need to be serious options, but this question is designed to get one response and conflates different issues.
There needs to be a balanced question about whether the provision of childcare is worth having, eg by asking whether the respondent would support increased childcare at the expense of some other item of expenditure or if it meant putting up taxes. (But even the latter is not very good as people tend to say they would support tax increases and improved services when it is a hypothetical question.)
Given the fiscal powers Holyrood has just now, especially in this era of cuts, providing additional childcare could only be achieved by cuts elsewhere – but the question as it stands would have us believe that John Swinney is holding back on this provision and will only grant it in the event of independence. Equally, the Scottish Government should be asked (and is) about how they will fund this and other commitments.
The only question I find remotely interesting here – though still flawed – is the preferred constitutional option question at the start.
So what we have to ask is why does a ‘gullible or biased media’ fall for it every time. Oh wait, I think l know…. key word here is biased but the media pretend, all bar the Sunday Herald which is doing a good job providing balance, to be gullible
And well done on getting Professor Mitchell to go over your questions and for his critique here. You really are providing quality Rev Stu.
Highly informative, but not overly surprising.
Thanks for allowing your reply to be published, Prof. Mitchell.
That, my darlings, is an elegant demollition
If we had ham, would you like ham and eggs, if we had eggs. Some questions are really nonsensical.
What YouGov should have asked is how much the Scottish Government is spending on Space Monsters to tweet death-threats to Susan Calman. That’d get a gullible press going.
Great stuff. It’s good of the professor to confirm what we already knew! There are some points there that would never even have occurred to me, so clearly the psephology stuff is a bit more complex than one might imagine at first.
I don’t agree that there would be a referendum on further powers if there’s a No vote, though – even ignoring the fact that they won’t devolve any more powers, I just don’t believe there would be a referendum on it. The whole point of devolution is to keep the process exclusively for unionist politicians, who decide amongst themselves and their little commissions which powers Scotland is allowed to have. Giving the public the chance to have their say risks us giving them an answer they don’t want!
Good stuff.
I’d be happy to chip in to fund a poll that asks these questions in the manner prescribed by Professor Mitchell.
If only the public could get to see this breakdown – as we have said time and time again why does the BBC keep going back to Prof John Curtice all the time. Some fair analysis or a fresh view here from Prof Mitchell.
Great work Rev.
Does anyone know if Larry Flanangan of EIS is on BT board ?
I looked on the ‘What Scotland Thinks’ website:
link to blog.whatscotlandthinks.org
and I could not find a critique of this poll by John Curtice. I wonder if he will be giving one?
Yep Doug, devolution only leads to an increasing vote for Nationalists and an increase in belief of the Scottish public in their own country…..as has been proven.
There is no way they will make the same mistake twice.
[…] On gullibility and bias […]
@jimbo:
That strikes me as a very good idea. Get Prof Mitchell to write the questions in an unbiased style, then have Curtice review them.
Only downside is that to see the clear effect of the question wording, YouGov would need to do the poll again. And frankly they’re a shower of shysters who don’t deserve our money.
Just watched PMQ.As soon as Scottish independence is mentioned by Ian Davidson all his Tory friends had a good old Guffaw and a laugh at our prospects. Hope the actions of people like Davidson etc are remembered after independence when they try to get in on the act.
@gordoz
Does anyone know if Larry Flanangan of EIS is on BT board?
I doubt it since he’s pro-Indy. Naturally, since the culture of the EIS is very Labour leaning, that tends not to be something that’s discussed much… They’re pointedly ignoring the topic at their AGMs.
You can find the members of the board here: link to companycheck.co.uk
No wonder the English think we are feeble minded and backward, When they see that clown Davidson running down his own country and people. He is the pin up boy of the TORY PARTY.
HOPE ALL THE NO STALWARTS ARE PROUD TO BE SCOTTISH!
@ Tamson
“Only downside is that to see the clear effect of the question wording, YouGov would need to do the poll again.”
Oh, I dunno. I don’t see why Panelbase could not ask the same questions posed in the proper manner. It could have a two fold result by showing up both YouGov and BT as inadequate.
Gordoz
Dorice, a male blogger pro indy and English, haunts The Guardian. He is very erudite and I think he may have published the odd comment here.
He said that Vince Cables’s Department has financed a number of academic opinion leaders on various subjects and thus they can be offered to the BBC at no cost. Prof curtice was one of them and I think also Prof Hugh Pennington, who is a Labour activitist.
That could explain why some faces seem to be the expert to go to by the BBC?
OT but after yesterday’s attempted revisionism over devo-max being rejected, this has to count as one of the most breathtaking history re-writes I’ve ever seen. Watch for this one becoming more wide-spread
Dougie Hopkins: All I would say is people need to vote, watch out for tactics like in the seventies, when no voters were told if you if you stay at home it would be classed as a no vote, the SNP then tried to claim victory because they got 2% more than the No voters on a 50% turnout. You need to vote this time
TheeForsakenOne – Many thanks for clearing up.
That makes sense from a response I got from him in an email. Its just someone else had posted on WoS, that he was a board member of BT ?
@Bugger the Panda – Aaahhh (thanks for the heads up)
I think we all know that the unionist polls are skewed.
O/T.
Want to ask Alistair Darling a question via the BBC?
If you have something to ask Mr Darling email newsonlinescotland@bbc.co.uk
Please put “Referendum questions” in the message field of your email.
Go for it folks.
“a survey designed less to find out what the public think than to try and get a gullible or biased media to provide a headline.”
I thought this was only for the published polls. They don’t let people know about the ones that gather useful information.
OT (sorry for the double-post):
A fun new game for Wedensdays:
Write Johann’s question for the FM for FMQs tomorrow.
The topic’s normally predictable enough (I can’t see what it will be this week though).
Maybe if we start getting it accurate enough she’ll start being more creative?
It’s refreshing to hear the views of other credible pundits. Mr. Mitchell wasn’t exactly in tune with the usual BritNat interpretations the are regularly broadcasted into our living-rooms, so perhaps that is why we rarely, if ever, hear the likes.
OT. I just found out that the Antisocial Behaviour, Crime and Policing Bill has reached its report stage. We’d better get it done this time, as it looks like our civil liberties are about to disappear.
Police State UK: At last, a law to stop almost anyone from doing almost anything
link to stratrisks.com
link to liberty-human-rights.org.uk
“A fun new game for Wedensdays: Write Johann’s question for the FM for FMQs tomorrow.”
I can’t think of an actual question but I predict a sneering reference to the fact that 59% of people in the Borders don’t want separation, with the poll or headline waved in the air for dramatic effect. Possibly a question along the lines of “the people of the Borders have made clear they want Salmond to stop his ridiculous obsession with separation because they realise we are better together better than anyone. Then probably a rambling sob story about some NHS patient who lives just south/north of the border and travels over it for care and how will the separatists be able to guarantee nothing will change for her?
Write Johann’s question for the FM for FMQs tomorrow.
a.k.a. What’s on BBC Scotland News today.
Anyone know the exact questions and phrasings from that borders poll btw?
Before clicking on the link for this post, I thought it might be news of the Ipsos-Mori poll on media objectivity that they conducted a month ago. I wonder if it will ever see the light of day.
Interesting comments by the Professor, and particularly so that these kinds of reservations about questions, which immediately met the first Wings poll, are NEVER seen applied to any other pro-union/MSM poll.
Funny that? No, not funny at all.
Off-topic, but here’s a video of Ivan McKee giving a presentation on the economic case for independence. It’s an excellent format for bringing the uninformed voter up to speed, I reckon.
Wow, that poll is pretty shocking.
Well done Rev and well done Prof Mitchell for an honest critic. I would suggest this explains the figures achieved in previous Yes/No unionist polls (as if anyone has any doubt) where they differ greatly from, say Panelbase. When’s our next poll, really looking forward to that
Illy says:
A fun new game for Wedensdays:
Won’t be able to answer until after the Labour Broadcasting Corporation’s six o’clock news surely. Doesn’t Paul Sinclair tell John Boothman what she’ll say the following day?
The borders poll, if it can be trusted, a useful tool for where YES need to direct their resources. Always an upside to these things!!
ComRes did the borders poll……they also do the independents monthly voting intentions polls….so probably westminster methodology
Johann’s Question
As someone who myself not only knows the meaning of motherhood, I would like to ask the [slidey eyes] first minister if he would care to make a comment on the value to the [barely choke back disdain] referendum debate of criticism such as that offered by [check notes carefully, roll as much eye as you can] Wings over Scotland on the conduct of the widely respected YouGov in drawing up their questions for their recent poll.
If we are to believe this website’s one so-called expert, polls nowadays are intended less to establish [big word, Cara pause] what the public think than [air quotes] “to try and get a gullible or biased media to provide a headline”.
Is this implicit desire for censorship of the world-famous Scottish free press and their polling organisations and advertisers something for which he would really wish to ask the mothers and potential mothers of Scotland if they would care to bring a child into the world? And just what does he know about motherhood, more’s the point?
I know I’ve banged on about this before, but as some of you may remember, I have been very keen to see a poll question which would fairly establish the support for independence if the economic argument, either way, was not an issue.
The reason for that, is that the economic argument is seen to be a key one and is thus subject to the most divergent claims between the Yes and No camps.
As such, a question like that, when the result was compared to the usual yes/no response, would give implicit information in regard to which side is currently winning the economic argument as well as a just how pivotal the argument was.
The problem has been one of formulating a question which both makes clear the mindset the respondent is being asked to adopt before answering the question and yet does so without being too wordy.
Examples I have given in the past have either been too wordy or unclear.
Here is my latest attempt, which while I think is clear, is probably (way) too wordy:
This question is designed to establish what the level of support for Scottish independence would be if the economic arguments were not an issue. With that in mind, regardless of your current opinion or if you have not yet formulated a firm opinion on the economic arguments we would like you to consider the scenario where by the day of the referendum vote you have become convinced that Scotland would be no better off OR worse off financially after independence.
In the circumstance described above, would you vote Yes or No in the referendum?
I would be grateful if
My my who’d have thought the unionist polls had questions that were skewed or biased in one form or another. The whole unionist movement is skewed.
P,S. (pressed send prematurely)
… that question could be assessed and rewordings suggested.
P.S.2 “by the day of the referendum vote” should have been comma’d .
I think analysis of responses which differ between this question and the usual straightforward question with no preamble (which should be asked first of course) would be most revealing.
For example, if there are more No’s which ‘convert’ to Yes’s than there are Yes’s which convert to No’s, that would infer the No camp is currently winning the economic argument.
The total number of ‘conversions’ would indicate how important or not the economic argument is.
Re Wednesday game-Gordon Brown has spoken,Johann will have had her instructions, put the frighteners on the workers of Rosyth, there’s a by election coming up.
Alexconn
Have emailed my question to the beeb to ask Mr Darling its
For Alistair Darling
As the person responsible for doubling the UK’s National Debt and raising the annual deficit to £175 Billion, what were your plans for repaying the debt?
And given your incompetence in financial matters , what gives you the right to lecture on financial and budgetary matters to an Administration that has balanced its books for every year its been in office?
This question is designed to establish what the level of support for Scottish independence would be if the economic arguments were not an issue.
Stop there. The rest of the Q is unneccessary.
I too would like a Q like that on a poll to see just how much the Indy Ref vote might be ‘from the heart’ driven.
OT: @alexicon
I found this short clip of Alistair Darling from the 80s (sorry its from the BBC) link to news.bbc.co.uk Darling gives his views on budget cuts and nukes.
Its completely hypnotic. I must have watched it at least 10 times. Darling looks like he’s on a Trainspotting comedown.
Chic McGregor,
here’s my stab at it, still quite wordy though I am afraid. Could possibly lose ‘even if not yet fullyformulated’ ?
This question is designed to establish what level of support Scottish independence would be at if the economic arguments were not an issue. Therefor, regardless of your current opinion, even if not yet fullyformulated, were you by the time of the referendum vote to believe Scotland would become no worse or better off economically after Independence, how would you vote?
Should Scotland be an independent country?
YES or NO
Chic McGregor,
Attention spans being what they are, I think it’s almost impossible to arrive at a question that wouldn’t be too wordy or throw people off the trail.
That said, how about
If, at the time of the referendum, it had been firmly established and generally accepted that Scotland would be no better or worse off economically as a result of independence, how would you answer the following question?
Should Scotland be an independent country?
YES or NO
Chic McGregor,
This question is designed to establish what level of support Scottish independence would be at if the economic arguments were not an issue. Therefor, were you by the time of the referendum vote to believe Scotland would become no worse or better off economically after Independence, how would you vote?
Should Scotland be an independent country?
YES or NO
@ Chic McGregor
I think what you are looking for has already been effectively answered : ICM Sept 2013
If Indy would make you £500 better off :how would you vote : YES 47% / NO 37%
link to whatscotlandthinks.org
If Indy would make you £500 worse off how would you vote : YES 18% / NO 66%.
link to whatscotlandthinks.org
I think WOS also asked the ‘everything else being equal’ question and got a NO lead of only 4%.
My view on it is that given there are huge leads for Yes on the more Scottish than British question and on the Do you trust Holyrood over Westminster then the only thing holding back the undecided from moving to Yes is the lack of understanding of the affordability issue.
Prof Curtis is forever preaching the same message – and while he may not be everyone’s cup of tea he calls the data as he sees it, and that’s what it tells him.
A question I would love to see asked in a Scottish poll.
“Is it normal for a country to give all its assets to another country to manage them for them?”
Or something along those lines, without mentioning Scotland.
Ivan McKee,
that was a great presentation on Scotland’s economic position in relation to the UK! I just watched you over on Bella and have already sent a friend of mine it. Just the kind of fact based reassurance he has been looking for to help him towards a firm YES. So thanks a lot and please keep it up.
Cheers min!
@Ivan McKee
Thanks Ivan, but I knew that. Past times I have advocated a full ‘Moreno type’ question which would enable a more detailed analysis. But that involves cost and complexities.
Its getting, a single question which is clear on the required mindset, concise and which is a supplementary to the standard unqualified question and which is adjudged fair and unbiased in wording.
If Indy would make you £500 better off :how would you vote : YES 47% / NO 37%
That’s not far off what the result will be; yes a little higher, no a little lower, but within error.
It won’t be down to economics though. It all comes down to national identity largely (there’s cross-over of course, but this is what its about in the end).
Of course people say it’s for all sorts of reasons; politics, trident, EU, left / right and the economy.
It’s because saying ‘I’ll vote for independence just because I’m Scottish’ can sound a bit silly. It’s not – it’s the reason just about every country exists – but it’s easy for opponents to jump on / make out to be irrational. Of course you might sound even more silly if you said you’d vote for independence if it made you worse off, so naturally, Yes is very low in this case.
£500 – the price of a modest spec PC or a week in Benidorm – quid gives you a great reason to justify your intention to vote Yes without fear of looking silly.
@WND
“If, at the time of the referendum, it had been firmly established and generally accepted that Scotland would be no better or worse off economically as a result of independence, how would you answer the following question?
Should Scotland be an independent country?YES or NO”
The problem I see with that wordage is that is not personalising the hypothetical conviction. What I mean is, that it is possible, for instance, to envisage some people acknowledging something is generally believed but not agreeing with it and therefore going with their own current belief.
Example. Someone who believes Scotland would be worse off so is resolved to voting No and would do so even if they knew the general belief was that it would not. i.e. they have more faith in their own belief. But if it is personalised by asking them to imagine that they themselves had been convinced by the time of the vote that Scotland would not be worse or better off, that is a slightly different mindset.
The trick here is to make clear to the respondent that they must put aside any current conceptions and envisage, as best they can, how they would vote were they convinced there would be no fiscal consequences to independence.
People are always going to be inclined to revert to current belief if they are given any kind of escape clause.
Moreno type’ question
The census question was better.
The Moreno question is problematic because it portrays British as ‘bad’ in the first option to an extent.
Scottish NOT BRITISH!
Is what Scottish not British says
Hence the 62% Scottish Only (i.e. not British) – as per the census – end up splitting between Scottish not British and More Scottish than British roughly equally. The ‘hardcore’ (for want of a better word) pick the first. The rest feel it’s a little shouty and could be portrayed as ‘anti-‘ so opt for the second.
Ivan,
Yes your video over on BFS I have shared with a lot of folk, including on here. Great stuff. Just what we need.
The first pollster to weight to the census identity question will be the one that’s the most accurate.
It is after all a referendum on national identity.
MORI have a huge problem with this; too British due to landline telephone. Add in the shy factor and skewed to No. It’s like asking all your respondents what they voted in the last Scottish election and 30% say Tory but you don’t take that into account for election voting intention. Instead you just report them as a representative sample.
Panelbase asked an identity type question and it looks like their base is much more representative.
I think the +/- £500 question clearly demonstrated the importance of the economy to the campaign, but was always going to be subject to the bribe accusation criticism.
Which in effect. made it difficult to use as convincing propaganda.
I think the +/- £500 question clearly demonstrated the importance of the economy to the campaign
Honestly, ‘the economy’ won’t make much if any difference. It is the best reason people can come up with to justify voting for something they’re maybe a little shy about admitting too.
So long as they are fairly confident the sky is not going to fall in, they’ll vote Yes just because in many ways.
Agreed it backfires though because you can turn it around as a ‘cheap bribe’ as you say. £500 is the sort of figure that just sits on the point of sounding like a lot of money. £100? Doesn’t really cut it. £250? Still sounds a tad small… £500? Well, that’s half a grand – it would be silly not to vote for such an improvement in finances! (whew, I can justify why I plan to vote Yes to anyone).
Scotland will become independent because it is Scotland. It has been losing it’s Britishness since it’s peak in the 1950’s post war consensus. The empire is gone and so is the consensus.
If Scotland was all British in identity, there’d be no SNP, no Scottish Parliament and no referendum. Identity is the driving force behind everything and will decide the result, just as it did in 1979 (when Scotland was not far off 55% Scottish / 45% British in identity) and 1997(75% Scottish if forced / 62% Scottish only).
Its the reason Denmark is not part of Germany. It’s the reason Norway and Sweden split. It’s the reason the empire is gone. It’s the reason it failed in Quebec; too Canadian.
Politics, economics, trident, the EU… These are arguments used to justify it. Nothing wrong with that – they can be good reasons. They would be irrelevant though without the fact that Scotland far more Scottish than it is British and Britishness has been in decline since the middle of the last century.
6-7/10 for devo max
6/10 Scottish only, 7.5/10 Scottish if forced to pick
3/10 always say yes
3/10 Scottish not British…
All about national identity.
@SS
Sorry, I wasn’t clear there. By ‘Moreno Type’ I only meant with a similar kind of 5 part sliding scale. e.g.
A lot better off.
Slightly better off
Neither better nor worse off
Slightly worse off
A lot worse off
Wordage apart.
I agree with your ‘Scottish, not British’ wordage crit but would suggest that the recentish, last ten years approx, wall writing campaign mainly in the Edinburgh area and probably started by some weel kent republicans, where they write ‘Scottish Not British’ on walls, has probably nuanced the phrase in a way that was not current when the question was coined.
@SS
“Honestly…”
Oh I agree with you there to a large degree. For all its flaws, even if only for shiting language nuance reasons, the Moreno Question exhibited exactly that, albeit not specifically on independence.
There were at least a couple of surveys where there was a supllementary straight choice question, Scottish or British.
In those instances, it was clear that nearly all of those who said they felt equally Scottish and British plumped for Scottish, when push came to shove.
Same kind of thing which we hope, with a degree of expectation, will come into play on R-Day.
Sorry, I wasn’t clear there.
No problems. I just found some of what was being discussed as an opportunity to waffle a bit. 😉
SS
Thanks for stopping me correcting to ‘shifting’ 🙁
Freudian slap 🙂
If you start from a basis that most people are reasonable (having sound judgement; fair and sensible).
If your fundamental belief is that a country should run its own affairs and you are able to demonstrate that it can.
Then surely, over time, those that oppose your standpoint will increasingly come across as unreasonable (not guided by or based on good sense).
This is what seems to be happening already and long may it continue.
I think it is important that we distinguish between expectation and campaign. Yes, I do think there will be a ‘referendum day effect’, I’ve said that for years. Its a unique situation, people will have a truly once in a lifetime chance to either change the constitutional status of Scotland or leave things as they would be under the UK. Very different from the psychology of polling, where it doesn’t really matter.
On the day, people will largely vote as individuals rather than in alignment with whatever polls are saying, at least to a far greater extent than is normal for recurring elections.
However, that does not mean that we should not maximise the information available to people before they walk into that booth.
Although much less than is normal for an election there will still be a significant portion who will still be influenced by poll results or whether they believe the fiscal scaremongering.
It is still therefore important to make sure people are armed with the true aspirational level of support for independence which I believe exists.
The one’s who will vote Yes and decide the Referendum, are totally off the political agenda. The people who never engage in the Political arena, and mainly despise politics. Their intentions can never be gauged and that screwed the Polls. The Unionists then publish abd congratulate themselves on the screwed Polls, like a morbid game. The piper calling the tune being taken to the cleaners.
In other words, a Unionist fix, of deluded proportions. The higher the turnout, the higher the Yes vote. Win, win.