If we must, then
The No campaign got itself rather excited today about the third independence poll of this week, this time by TNS-BMRB, which showed a spectacular and unexpected doubling of the “Don’t Know” figures at the expense of both Yes and No.
We didn’t go into the other two in any depth (noting only the difference in media coverage of them), because as we’ve said for the last 18 months, simple Yes/No polls at this stage are fairly meaningless. But this one deserves a little scrutiny.
Here are a few highlights:
1. The data was actually weighted in favour of people who aren’t going to vote in the referendum. Those “Certain” or “Very likely” to vote were downplayed in favour of those “Not very likely” or “Certain not to vote”. We’re not sure what the rationale for that was.
2. Similarly, it was weighted to over-represent the people who didn’t vote in the 2011 Holyrood election. More remarkably, the sample – in both weighted and unweighted varieties – suggested that Labour actually won the 2011 election, with more Labour voters polled than SNP ones.
A whopping 11% couldn’t remember who they voted for.
(If you weight the sample properly, ie according to the actual 2011 vote shares, the Yes/No vote comes out almost neck-and-neck.)
3. The poll recorded people who said they were certain not to vote in the referendum, but then included their opinions on the referendum question anyway. People planning to not vote No outnumbered those planning to not vote Yes by a huge 53% to 4%, with 42% unsure which option they weren’t going to not vote for.
There was also a huge margin for No among people “Not very likely” to vote, compared to much closer numbers among people who are planning to vote.
We imagine that Yes Scotland will be distraught to learn that a huge majority of the people planning to stay at home on 18 September 2014 are people who would have been voting No, if they’d voted.
4. The poll found just 16% of residents of the SNP stronghold of Mid Scotland & Fife planning to vote Yes – a mindbogglingly unlikely figure compared to 29% in the South of Scotland (traditionally the most anti-independence region) and 28% in both Lothian and Highlands & Islands.
(Above left: the 2010 Westminster electoral map of Scotland, next to the Holyrood 2011 one, both showing fertile SNP ground in Mid/Fife, and the Borders dead zone.)
Do we need to go on?
The data were?
OK, so shoot me.
They’ve weighted using the wrong election. Have they?
I noticed that John Curtice took this one with a huge pinch of caveat salt too.
How on earth could i in 10 not remember who they voted for? Were the polls conducted in the retirement homes for the bewildered?
Who commissioned this one by the way? Can we guess?
So in essence, a pile o’ kack . I bet their clients are happy though. Can you imagine any neutral or pro indy client agreeing to that methodology? Therefore the clients must be seeking the result they want by whatever methodology is most useful to them.
Maybe the soon to be commissioned poll by WOS should include this question;
“Do you place any importance on opinion polls prior to voting?”
@ HandandShrimp
Postal voters!
All TNS have done is embarrass themselves and have as a result suffered reputational damage.
The data is irrelevant beyond using it for a laugh. The British media has found its voice again by publishing any old rubbish it can use to “prove” that the SNP/YES is going to fail & that voting for Salmond’s vision is a waste of time.
Having almost run out of scaremongering rubbish to print, they can always fall back on flawed polls to get their message out to the die hard idiots who can always be relied upon to believe any old shite the Scottish/British media prints.
It’s all bollocks. We all know the YES vote is sitting around 40% with 20-30% DKs and it’s been like that for months with the NO vote gradually melting away. That’s why Better Together are always bricking it.
If it helps at all, I volunteer to do a poll of one person who is full of Lambrini.
ianbrotherhood–that surely counts as ‘self selecting’ ?
I know we all have anecdotal evidence of the way things are going but when I see polls from both sides of the fence all I can see is bollocks.
I’m struggling to find anyone at all who’ll be voting no next year (putting aside the scarily inept councillors & their fuckwit lackies on twitter). Even workmates who are voting yes are reporting their friends and families are all voting yes.
Maybe I just live in some kind of yes bubble but my gut feeling is that this is going to happen despite what the polls & the unionist TV pundits are telling us.
Let’s get this gig done folks!!
This was a poll that was meant shape public opinion and not to reflect it. It was crudely dishonest and deserves to be mocked. TNS have made a right erse o’ themselves.
“We all know the YES vote is sitting around 40% with 20-30% DKs and it’s been like that for months with the NO vote gradually melting away”
How do we know that?
Going O/T here so apologies.
I listened to Radio4’s ‘Media Show’ this afternoon and they had a section on how the media, particularly the BBC were covering and going to cover the referendum.
Contributions came from Lesley Riddoch, BBC Scotland’s John Boothman and The Mirror Groups Alan Rennie. All in all it was a pretty disappointing affair – the usual platitudes and denial from Boothman that BBC Scotland are doing anything wrong and it finished up with the presenter Steve Hewlett asking each of them how they were going to vote. Riddoch said Yes, Rennie said No and Boothman said Yes/No.
When you take this in context of Director General Tony Hall speech at the Edinburgh Media Festival the other week when he announced it would be difficult for the BBC to be impartial over the referendum, you realise this discussion barely scratched the surface of this subject and that once again licence fee payers will be fobbed off with more meaningless ‘look nice’ self-investigations by the BBC into their own output.
If you want to listen to it it’s available here on the BBC i-player and starts at around 10mins into the show.
link to bbc.co.uk
It’s probably impossible but what we need is a poll commissioned by a party that does not have a foot in either camp and that is carried out by an independent regulated polling company that polls a genuine cross-section of the population and doesn’t manipulate the weighting to suit the results it (or the commissioners) want, and that isn’t carried out by a polling company which only polls people who actively sign up to it and which can therefore be manipulated by a concerted campaign from a particular interest group to get their people to sign up for the polling and therefore skewer the results of the poll.
Aha, another poll by TNS a.k.a. The NO Supporting poll!
What’s interesting is that people who are absolutely certain not to vote on something still voice a clear opinion on it.
I’m wondering if that group was dominated by labour MPs.
and therefore skewer the results of the poll
Shish kebab style you mean?
Best thing TNS could do with this one.
PDSA – by Jupiter that’s a long sentence.
By the way, why don’t you have whip-round your Punic conquests or the Numidian nether regions and organise a poll yourself? Of course the ‘elephant in the room’, if you allow the historical allusion, is that you probably Cannae.
It can happen, you get a “bad” sample. The only thing to do is publish and be dammned. They say this is the first in a series so lets see where the series takes us.
Why do I think it bad? That Labour bit of mid Scotland and Fife with Helen Eadie MSP and Dr Brown MP we’ve had nights with only 15 out of 60 Yes but then again we’ve had 25s and the Nos have ranged 18 to 30 so that 16%, though possible, seems improbable.
@ Rabb – I couldn’t guess the percentage of YES! intentions nationwide but I think you’re probably right. Most folk I know are either definte or swithering YES because they cannot take any more of the drivel we have from Westminster and from the official labour in Scotland line.
Whits this PDSA thing. The PDSA done an emergency historectomy on ma pals staffie. They are a brilliant organisation Major, you really need tae stop mocking them.
Apologies for going OT but here is another benefit of being in the Union.
Swansea and Stirling’s elite swimming centres under threat
link to bbc.co.uk
I must try to think about what I’m saying first.
cartwheel 
Of course having this as the starting point for a series is brill. What trend is the next one going to shew? I should have thought of that first. Of course this poll is absolutely 100% kosher. It even satisfies PDSA’s requirements.
Even John Curtice urges “caution” re. this poll.
@Major – That’s a nasty mind worm you’ve let loose there.
PCSA not PDSA.
@sneddon-
re ‘self-selecting’.
I am not at liberty to divulge the identity of the person being polled – it may or may not be me, but I’m not sure, and even if I was sure I wouldn’t be allowed to tell you or anyone else.
@handclapping
(Just my little joke.)
A few points:
1. The data was actually weighted in favour of people who aren’t going to vote in the referendum. Those “Certain” or “Very likely” to vote were downplayed in favour of those “Not very likely” or “Certain not to vote”.
It’s probably a side-effect of weighting for age. Older people are more likely to agree to be interviewed and also to say they will vote (see page 5 of the TNS tables). So when you balance up the sample for age you’re also reducing the likelihood to vote a bit.
One interesting point about likelihood to vote though is that the TNS press release highlighted both the general referendum results (Yes 25%, No 47%, Don’t Know 28%) and those for those Certain to Vote (Yes 30%, No 51%, Don’t Know 20%). It’s not a massive difference but the media picked the more unfavourable one to Yes, though most pollsters would take LTV into account in which findings they reported.
2. Similarly, it was weighted to over-represent the people who didn’t vote in the 2011 Holyrood election.
That’s probably OK, especially if you expect the turnout to more in 2014 than the 50% in the Holyrood election.
More remarkably, the sample – in both weighted and unweighted varieties – suggested that Labour actually won the 2011 election, with more Labour voters polled than SNP ones.
Yes, very weird. The sample has 21% SNP, 23% Lab, 6% Con and 7% Lib Dem while the actual electoral figures are 23%, 16%, 7% and 4%. It’s too big a discrepency to be explained by chance. It could be false recall with normally-Labour voters who chose SNP misremembering, but the YouGov poll seemed to show the SNP still doing very well, so it’s not the usual “spiral of silence”. TNS are going to have to look very hard at this.
A whopping 11% couldn’t remember who they voted for.
IT’s probably because they didn’t vote , but couldn’t remember if they did or not, rather than forgot which Party. With elections three Mays in a row, the not-particularly-political may get confused.
4. The poll found just 16% of residents of the SNP stronghold of Mid Scotland & Fife planning to vote Yes – a mindbogglingly unlikely figure compared to 29% in the South of Scotland (traditionally the most anti-independence region) and 28% in both Lothian and Highlands & Islands
Most of this will be because of the small sizes of the sub-samples of 130 per (most) regions – margin of error will be +/_ 8 points, that’s a lot of variation. But maps are misleading and Scotland is more uniform than you think. In 2011 the SNP regional vote only varied between 39% (Lothian) and 53% (North East). Mid & Fife (45%) was only 4 points high than South (41%).
I feel sorry for the ‘poor bloody indians’ at TNS as this cannot help their continued employment prospects. For a ‘reputable’ pollster to tie themselves to the mast of a sinking ship makes the P45 scenario more likely. Somebody will get an OBE though!
Results of this rubbish made big news front page on the Telegraph this morning.
YES VOTE SINKS TO 25% !!
What a load of crap!
“a huge majority of the people planning to stay at home on 18 September 2014 are people who would have been voting No, if they’d voted.”
Well just goes to show that the endlessly dire warnings of Scotland’s destruction have sacred the No voters into staying in their homes. Maybe we could term this as “friendly fire”; blue on blue?
Serious question – has any reputable polling organisation ever gone the whole-hog and connected people to lie-detecting contraptions before asking them anything?
And if not, why not?
‘Nothing to hide, nothing to fear’ an’ aw that jazz…
tartanfever says:
“I listened to Radio4?s ‘Media Show’ this afternoon and they had a section on how the media, particularly the BBC were covering and going to cover the referendum.”
Thanks for that tf.
Boothman, “BBC Scotland is covering this well” (the independence debate)
O’ wad sum powr the gift tae gie us, tae see oorselves as ithirs see us!
Boothman, “If we’re not bold, then why in the last six months we have been to Spain, Ireland, Denmark, the US and Canada and other countries to cover this story?”
As I recall John, Spain was for wee Glen to try and get people from the EU to say Scotland couldn’t/wouldn’t be part of the EU post independence, likwise all the other destinations (I can’t be bothered listing, you all know..)
It must take some nerve, it’s not as if he doesn’t know what he is doing.
Because nobody would agree to it? And because there is no such thing as a functional “lie-detecting contraption”?
FFS, TNS, PDSA, OBE. Whits going on here . Ah canny make head nor tail of this tonight. Romans and bloody artificial panda insemination, ach I’m away tae watch big brother with the missus, the folk on there are crackpots but you lot need tae take a good look at yer selfs in the mirror.
@Morag-
‘Because nobody would agree to it?’
They could be offered a small inducement e.g. a tenner?
‘…there is no such thing as a functional “lie-detecting contraption”?’
A Bible, Koran, Torah etc etc?
AlexMcI- I did look in the mirror and I’m beautiful man
“Ones who couldn’t remember who they voted for”. That was probably Labour MSP’s.
Seriously this has taken desperation to a different level. They role out Gordon Brown. A man whose ego massively exceeds his trustworthiness.
Sorry for quick off topic, just back from the Yes Kelvin launch, great speech from Dennis Canavan but Jeane Freeman was especially brilliant.
Really good turnout, about 170 folks.
@Alex McI
Alex – is that why they’re having problems at the zoo – they’re trying to inseminate an artificial panda?
@ Jeannie, my wee dog had a bash at that with the wee ones toy panda. Alas it didn’t work and missus Alex keeps telling him he is a dirty wee boy.
who knows they might have more success at the zoo, although I doubt it.
@Sneddon, that’s not a mirror by the way, it’s a life size jackie poster, and you need to get a change of prescription mate lol.
@Alex McL-
‘Jackie’ and ‘Jocky’ are easily confused…
You’ll have seen this, but it’s always worth running out again. (For anyone who hasn’t seen it, check out the face on the big screen behind the performers.)
The two straw polls Ive carried out have both small samples and wildly varying results. The first was in a pub with some mates from all over Britain, 3 from London, one from Leicester, one from Durham, the otther 5 from various parts of the country. We had quite a few braveheart , shortbread too wee too poor arguments thrown about, but the upshot was 100% yes of eligible voters, discounting a devomaxer, who says he is forced to vote yes. The other poll was in my office where it is 75% No, with most Nos being soft (FUD) and one who thinks independence will be a total disaster!
I do wonder if there’s a long-term effort going on here, to discredit the notion of opinion polls on the referendum altogether.
Firstly, the MSM now realise the genie is out of the bottle on “control” over opinion polls. If this site can crowdsource two quite easily, what’s to stop others doing it?
Secondly, I read somewhere that the history of independence referendums shows a consistent rise in Yes votes as polling day approaches. Perhaps by planting and nurturing the “Don’t believe opinion polls, they’re all rubbish you know” seed now, they hope to stop any “surge” during the campaign.
If I understand it correctly, the SNP Panelbase poll asked questions that led the responder to consider the effects of policies.
So, the objection is, if people are given a chance to think about what Indy might mean, they are tempted to vote for it, and that will never ever do!
Opinion polls are interesting and useful for showing trends. Only a fool would blindly base their entire strategy on them. It amuses me massively, however, that the unionists are only attacking them when they have unfavourable results
If the 59% voting No number was correct, that would mean more Scots are satisfied, trusting and confident in Westminster politicians and Government than most of the UK.
@ Roger
Couple of points : The figures for those certain to vote are N 49% ; Y 30% (Not N 51; Y 30)
If you correct for actual 2011 SG Election results then the Yes share is increased by about 6% (mainly at the expense of Don’t Knows, but also some reduction in the No %).
This driven by the 41% difference in Y intentions between SNP voters and SLAB voters across a 15% error in the relative sample sizes.
This brings you to N 47%; Y 37%.
Or excluding Don’t Knows N 56% ; Y 44% : Which I think is more in line with where we actually are at the moment.
If you are in Scotland, and talk to people, you’ll know just how soft that 56% No vote is.
Anyone think the Herald will comment on error of this Poll being misleading front page news ??
I’m not sure I understand the rationale for weighting polls at all. The purpose of a poll is to weight CURRENT opinion. The weighting idea seems to designed to check whether the sample is “representative” and re-weights numbers according to a previous election. But this is circular. It biases the current result based on a previous result. As far as I am aware, when we go to the polls our votes are not counted then re-weighted on the basis of what we voted last time.
If there is a concern that a sample may be biased then minimise the risk by choosing a larger sample and being careful about the selection methods but don’t adopt a methodology that deliberately biases the current result towards the previous one. So what do raw, unweighted Y/N figures come to?
Never mind what you told the last polling person who called you.
Vote early, vote often, vote YES.
You know it makes sense.
GP Walrus
Agreed. Any weighting process is open to abuse.There should be a universally agreed set of conditions -ie representative cross sample on gender, age, social group etc – before any poll is considered accredited. But that wouldn’t suit our enemies.
What you have voted for or supported in the past should have no influence on how your choice is currently measured
@Dave McEwan Hill
Yes I agree.
Daughter (13) in leau of going to sleep way past her bedtime has just announced that if she were old enough to vote she would definitely be voting yes. six months ago she was no all the way!
All I can say is “If you cannot remember who you voted for” then you are so brain dead you should not actually be allowed to vote.
I can’t remember who I voted for in the two 1979 elections. I’d like to think it was the SNP but I don’t honestly know. I know I voted Yes in the devolution referendum without paying a lot of attention, because as far as I was concerned it was simply a no-brainer, and I know that by 1981 I was a confirmed SNP supporter, but the Westminster elections of 1979 are a blur.
Ivan McKee says:
Couple of points : The figures for those certain to vote are N 49% ; Y 30% (Not N 51; Y 30)
Actually not. You’re looking at the combined figure for Certain and Very Likely to Vote (see p 4). The table is rather confusing.
I’m not quite sure how you did your re-weighting but I adjusted each figure (Y/N/DK) for each Party to what it’s 2011 percentage was. I also assumed the Can’t Remembers were really all Did Not Votes and also assumed that all 2011 voters would vote but that only half the CRs and DNVs would. This would give a turnout of 75% which seems reasonable.
The result I got was Yes 27%, No 45% DK 27%, so weighting to 2011 makes some difference, but not as much as you might think. This is because the only sustantial change was reducing the Labout allocation and also because TNS found only 55% of SNP voters who would vote ‘Yes’. I suspect that using only Certain and Very Likely To voters would add a few percent to Yes though.
And do you really think it’s at all likely that only 55% of SNP voters are going to vote Yes, in this space-time continuum?
Exactly, Morag
The whole poll is a concoction.
I’m at this on the front line every day. The change is going on and speeding up but at the moment mostly with the well informed. If this continues the others will follow. What is very significant is the huge difference between YES voters and NO voters. Most NO voters are not happy or open about their position. I venture to say “guilty” about it even. Most YES voters are upfront and confident and enthusiastic. The difference between public reaction to the two camps was also very pronounced at the Cowal games on Saturday. We had a continuous stream of wellwishers and ONE only offensive nutter in all day to our shop. The NO table up the road was subjected to a variety of abuse till they chucked it early afternoon. (I have to say I would expect the vast crowd at Cowal Highland games to be largely on our side however.)
114 (11%) can’t remember who they voted for, but are suddenly bothered enough to take part in a political opinion poll? Mmmm.
I think they know very well who they voted for…..
@ Roger, thanks for the response.
Take your point on the Certain / Quite Likely.
Its a matter of debate whether Quite Likely are going to vote or not.I had assumed it was Quite likely.
As you say it depends on what you estimate the turnout is going to be. (I suspect a lot of Turnout drop on the day is due to people who had intended to vote but don’t – due to holiday; death; sickness; house move; or critical family / work things that come up on the day – that applies equally to Quite Likely as it does to Certains).
Regarding the impact of 2011 voting intentions what I did was to take the SG2011 Actual voting percentages (45.4%; 31.7%; 13.9% 7.9%) and multiplied those out by the % for each party intending to vote Yes in this poll (55.3%; 13.9%; 5%; 11%) to get 32% Yes. Doing the same for No gets 47%.
So the SG2011 effect increases Yes by 5%; the Certain / Quite Likely to Vote effect also increases Yes by 5%. That would suggest that the combined effect of both could be close to 10%. (There doesn’t seem to be any tables correlating the 2014 Likelihood of voting with the 2011 Party vote so I don’t see any way of verifying that. I do accept that the data would suggest that SLAB voters are more likely to be in the Unlikely to Vote column so in this case 5+5 might only equal 8 or 9 and not 10, but its still a significant uplift)
Overall I think the polls (Panelbase; TNS and YouGov) are a lot closer to each other than the headlines suggest. Taking into account all the different factors : Margin of Error (3% ?); Question Sequencing ( maybe 3%, is there any evidence that says is makes any bigger difference than that ?);plus all the weighting issues in TNS and YouGov then I think we are at Yes in the low 40’s ; No in the high 50’s.
@ Roger
Apologies for Typo : Everywhere where I said ‘Quite Likely’ in the above post of course I meant ‘Very Likely’.
We have been out canvassing two evenings a week since June. The certain Yes vote is solid and probably increasing. It is at most a few points behind No and there is 30% undecided. Those undecideds want more information on the implications of independence, and they know they will not get that info from the press or BBC; without prompting they know the media are biased to the No side. They are almost all potential Yes voters, but YesScotland/SNP has to get its message across to finally convince them.
I’m certain the YouGov and TNS polls are flawed. We’re on track.
If it’s not too late, Rev, and after the announcement from GMB about cutting funding to the labour party, what about a question in your next Panelbase poll asking respondents whether they are a member of a trades union?
You never know, the results may help the unions decide where there political funding should be going, if they want to represent their members.
“If it’s not too late, Rev, and after the announcement from GMB about cutting funding to the labour party, what about a question in your next Panelbase poll asking respondents whether they are a member of a trades union?”
It’s not too late – I’m going to hold the poll back for a wee while, given the current ridiculous three-in-a-week rate they’re coming out at. It’s worth thinking about, certainly.
“How on earth could i in 10 not remember who they voted for? Were the polls conducted in the retirement homes for the bewildered?”
I suspect the usual SLAB trick of filling out postal votes may have something to do with this.
@ Ian Brotherhood:
In vino veritas – ply them with booze then ask their voting intentions.
Ivan McKee
The Certain/Very Likely To Vote thing was mainly me being pedantic, and in practice I think nearly all of the VLTV will vote too given the expected turnout[1], which should be more than the 63.8% of the 2010 Westminster election and certainly much more than Holyrood 2011 at 50%.[2] TNS shows 76% for CTV or VLTV in this survey.
But the same thing undermines the way you did your adjustments, because you effectively ignored the 50% who didn’t vote last time – and clearly a lot of them will vote. I gave then half value, which is a guess of course and maybe a bit high.
You also can’t then add on extra points for Yes supporters being more likely to vote because effectively you have already taken that into account by just using those who voted in 2011 – over 90% of them were CTV or VLTV (see pdf page 7 of the tables) and you can have them voting more than 100%. And the Yes increase you get get when you look at CTV is mainly due to DKs being less likely to vote – the No percentage will go up too if by less in proportion.
Using my rough adjusted TNS estimate of Yes 27%, No 45% DK 27% and then applying their CTV adjustments would give Yes 32%, No 51%, DK 17%.
All the pollsters use different methods and methodologies which is a good thing because we can see what effects different ways of doing things have. There is actually quite a lot of consistency within the results produced by individual pollsters – unlike say on an EU referendum where public opinion does seem to swing about a lot. So when you get what appear to be widely differing results, as this week, the underlying picture may still be the same and the discrepencies partly methodological, but also because pollsters may be measuring subtly different things.
[1] One of the oddities about the TNS poll is they ask the LTV question after intention in the same way the SNP/Panelbase one did (though the regular Panelbases don’t). I think this makes people who have ready said Yes or No more likely they they will vote, even if they are not really that interested. The other way round you get the ‘meh’ crowd more accurately
[2] One of the reasons that the regular ST-Panelbase polls are consistently more pro-Yes than others is that they filter by 8-10 LTV for Holyrood for the Referendum as well (this removes about 17% of the sample looking at May’s figures). But the 1 in 8 who vote for Westminster not Holyrood are much more likely to be No voters than average – they’re showing what they think is important by that choice – and they probably will vote in the referendum so they can keep on voting for Westminster.
Roger
I’ve been busy today, on and off flights, and I’m out tonight.
I’m going to sit down tomorrow with a clear head and a fresh spreadsheet and get my head around this and get back to you if that’s OK.
Appreciate the input and want to fully understand this.
Ivan
Bet MI5 Ghsq and secret service know who is going to win, thay know everything we think.
@ Roger,
OK I see what you are saying.
There is a double count effect if you include both the Weighting by SG2011 AND the impact of only considering CTV/VLTV.
So you are back to N51 / Y32 or N61 / Y 39 if you exclude the Don’t Knows.
Which is pretty much where everyone except Panelbase has been for months now, so the story really is nothing much has changed.
Ivan
Yes, despite all the “The polls are all over the place” narrative we’ve had in the last week, the truth is that the various pollsters are still showing internal consistency. This suggests the underlying public opinion on the referendum this year has been unchanged (unlike say the EU referendum where it’s gone all over the place). It almost certainly won’t stay that way, but the changes haven’t started yet.
Meanwhile Lord A is promising more polling goodies on Monday and says he has “completed one of the largest poll on Scottish politics”.