The emergence of policy-driven studies

Abstract

In the mid 1990’s, evidence emerged that the bicycle helmet law had failed to reduce the risk of cycling injuries. Governments responded by commissioning “studies” exaggerating the benefits of helmets while ignoring their tendency to increase accidents. Such policy-driven studies are quoted in official government communications to justify its controversial policy.  

These deceitful practices waste precious resources while real cycling safety is neglected. This might explain why Australia has one of the worst cycling safety record among developed countries, with a fatality rate 5 times greater and a serious injury rate 22 TIMES higher than best practice.

 

The hidden side of scientific research

Generally, research is conducted with scientific discipline, with a purpose to further scientific knowledge. However, sometimes the entity funding the research has a vested interest in a certain outcome, compromising the independence of the research.

This has been the case for some research funded by drug companies for example, where the research results are used to sell or promote drugs. Powerful government and corporate interests are increasingly influencing scientific research through controlling its funding, as reported by Brian Martin from the University of Wollongong:

“In the routine practice of scientific research, there are many types of misrepresentation and bias which could be considered dubious. However, only a few narrowly defined behaviours are singled out and castigated as scientific fraud. A narrow definition of scientific fraud is convenient to the groups in society — scientific elites, and powerful government and corporate interests — that have the dominant influence on priorities in science. Several prominent Australian cases illustrate how the denunciation of fraud helps to paint the rest of scientific behaviour as blameless”

Government is a large source of research funding. In the mid 1990s, evidence emerged that the helmet law had failed to improve safety. Several researchers reported this.  A bicycle activist who promoted helmet wearing and initially believed in the helmet law looked at the data and concluded:

“It is fair to say that, so far, there is no convincing evidence that Australian helmet legislation has reduced the risk of head injury in bicycle crashes.”

Governments responded by commissioning “studies” defending its policy, then quoting such policy-driven studies to justify its policy.

The Henderson report: a policy-driven study

In 1995, the Motor Accidents Authority of New South Wales (Australia) commissioned a policy-driven study that became known as the Henderson Report.  Written in an authoritative style, it pretends to be scientific. That illusion breaks down when one notices two unusual features for a scientific report:

  1. It contains many strongly worded, unqualified assertions.  Researchers with scientific integrity tend to be very careful with their assertions, making sure they are backed up by sufficient evidence and qualified by their context.  You rarely see unsupported or unqualified assertions in a scientific report.
  2. Assertions are not annotated, making it impossible to verify the data source.
This is not a scientific report.  It is unsubstantiated opinions presented as facts.  Here are a few of them (From the Executive Summary, under “The effectiveness of head protection”):
  1. “Helmet design and construction is based on known mechanism of head and brain injury”.  Not true.  This ignores rotational acceleration, the main cause of brain injury.
  2. “At the very minimum helmets halves the risk of head injury”.  Not true.  Helmets can aggravate brain injury through rotational acceleration.
  3. “Those who do not wear helmets are several times more likely to sustain injury to the brain tissue”.  Not true.  Ignores that helmet wearers are more likely to have accidents.
  4. “in Victoria, the number of bicyclists with head injuries decreased by 48 per cent”.  Misleading.  Ignores the decline in cycling and head injury reduction for pedestrians.
  5. “The vast majority of head impacts occurring … are easily survivable if a Standards-approved helmet is worn”.  Not true.  Helmets are not designed to protect in a serious accident.
  6. “No studies have come to conclusions contrary to the above”.  Not true.  The Hillman report (1992), one of the most comprehensive and famous review of helmet research at the time, is ignored.

Many assertions in this report are false or misleading.  The report ignores evidence that does not support helmets.

This “study” has received much criticism, notably in the vehicular cyclist:

“Individuals and organizations zealously pushing mandatory helmet use for cyclists are continuing to churn out reams of propaganda. One of the more voluminous efforts is Michael Henderson’s “The Effectiveness of Bicycle helmets: A Review” 1995, a politically motivated paper prepared on behalf of the Motor Accidents Authority of New South Wales, Australia apparently in a desperate effort to justify the State’s botched law outlawing cyclists who ride without a helmet.

Henderson’s report recycles much of the same old material that’s been cited by others over the years. The studies he references fail to provide a real world context, and to show any particular understanding of cycling. Although presenting bicycle head injuries as a worldwide problem, Henderson neglects to provide us with any sense of the size or scope of it.”

Contrast the bold narrow claims from this report with the more comprehensive approach from the Hillman report:

By wearing helmets, cyclists are at best only marginally reducing their chances of being fatally or seriously injured in a collision with a motor vehicle which is the predominant cause of these injuries. Even the most expensive ones provide little protection in these circumstances. Moreover, the argument in favour of helmets would have validity if there were proof that behaviour does not change in response to perceived risk. But there is no such proof. Safety devices encourage higher levels of risk-taking. As a result, cyclists are likely to ride less cautiously when wearing a helmet owing to their feeling of increased security. After all, the message of the advocates of helmet wearing is that such a practice will protect the cyclist’s head adequately in the event of any accident, not just a minor one when cyclists are hit by very slow-moving vehicles or fall off and hit their heads on the ground. Cyclists may be less likely to have an accident if they are not wearing a helmet, and are therefore riding with greater care owing to an enhanced sense of their vulnerability.”

Subsequent policy-driven studies

Official evaluations of the helmet laws commonly employ biased selection of research and statistics, resulting in benefits being unduly attributed to them and adverse effects (like an increase in the risk of accidents) ignored. Despite data showing that cycling safety has lagged pedestrian safety since the helmet law,  policy-driven studies found a way to claim that the helmet law was a success. For example, a 1997 report from a government agency made two misleading claims:

  1. “Cycling casualties decreased after the helmet law”.  This ignored the decrease in the cycling.  Per cyclists, cycling casualties increased.
  2. “A ‘strong correlation’ between higher helmet wearing rates and lower casualties”.  The underlying data indicates the opposite.

The helmet law was introduced as a part of a package of road safety measures including a crackdown on speeding and drink driving. The number of cyclists reduced significantly. Any assessment of the helmet law must take into account these confounding factors. Yet many government-funded “studies” like this one did NOT adjust for this, attributing all apparent improvements to the helmet law. Such negligence is difficult to comprehend. How could the “researchers” miss such basic adjustments? It is odd that these mistakes favor the legislation while the government funds the “research”.

Most medical case-studies, claiming that the helmet law has been effective, are also riddled with errors. They typically start from the assumption that helmets save lives and attempt to “prove” it by selectively fishing for data that supports their predetermined conclusion.

Some government studies not only contain false claims, but fail to rectify them after being corrected.

In 2000, the Australian Transport Safety Bureau (ATSB), a federal government agency, released a meta-analysis, that claimed to provide

overwhelming evidence in support of helmets for preventing head injury and fatal injury“.

This claim was rebutted in 2003, highlighting that:

“the meta-analysis … take no account of scientific knowledge of [brain injury] mechanisms”

The ATSB did not reply to the rebuttal, thus giving up on its claim. Despite being discredited, this analysis is still used by the government to defend the helmet law, claiming that helmets reduce the risk of head injury by 60%.

In 2011, a meta-analysis re-assessed this ATSB meta-analysis. It concluded:

This paper … was influenced by publication bias and time-trend bias that was not controlled for. As a result, the analysis reported inflated estimates of the effects of bicycle helmets …

According to the new studies, no overall effect of bicycle helmets could be found when injuries to head, face or neck are considered as a whole.

Publication bias is the tendency of contradictory or inconclusive results not to be published, resulting in a literature formed of apparently consistent findings that exaggerate the actual effect. Time-trend bias is the tendency to pick a specific time period that exaggerates the actual effect.

Since then, various government agencies have wasted more taxpayers money by commissioning policy-driven “studies”, attempting to obfuscate the failure of the helmet law.

A prominent university seems to be moving away from policy-driven studies, as reported here:

“the faculty wished to move away from conducting sponsored policy studies for government because this did not generate valuable intellectual property

The dire consequences of this dishonest behavior

Independent researchers who have studied the result of the helmet law have come to different conclusions.

Dorothy Robinson, a researcher from the University of Armidale, said:

mandatory bicycle helmet laws increase rather than decrease the likelihood of injuries to cyclists …

Having more cyclists on the road is far more important than having a helmet law, for many reasons …

[the] governments [which introduced the helmet laws] do not like to admit they’ve made mistakes”.

Bill Curnow, once a  scientist from the CSIRO, wrote as a conclusion in a scientific article:

“Compulsion to wear a bicycle helmet is detrimental to public health in Australia but, to maintain the status quo, authorities have obfuscated evidence that shows this.”

An independent group of public health and transport practitioners and researchers wrote in their report:

The failure of mass helmet use to affect serious head injuries, be it in falls or collisions, has been ignored by the medical world, by civil servants, by the media, and by cyclists themselves. A collective willingness to believe appears to explain why the population-level studies are so little appreciated. …. The disconnect between received wisdom and the facts is stark.

These are strong words from independent researchers, revealing frustration at the government unwillingness to admit they made a mistake.

For how long can policy-driven studies sustain failed policies?

In what way can this be justified as good use of taxpayers money?

Such deceptive studies add little to scientific knowledge. They exaggerate the reduction of minor injuries while ignoring the increased risk of serious injuries. Such deceitful practices have dire consequences. They tend to mislead policy makers towards false “solutions” to cycling safety, while neglecting more effective measures, like reducing the risk of accidents.

This might explain why Australia has one of the worst cycling safety record among developed countries, with a fatality rate 5 times greater and a serious injury rate 22 TIMES greater than best practice.

The practice of commissioning dubious studies to defend its failed policy while neglecting more effective safety measures constitutes an abuse of public trust among the bureaucrats entrusted with cycling safety.

Research on helmet law in the UK

In the UK, the helmet law has been debated for a long time, with the usual set of emotional arguments from helmet believers.  So far, rationality has prevailed, thanks to the efforts of the Cyclists Touring Club and independent researchers.  Here is some of the research that has help sanity prevail.

 

The Hillman report, one of the most comprehensive early research done in 1992.  It was ahead of its time.  Much of what was described in the conclusion subsequently happened in Australia and New Zealand.

The insightful conclusions for the report include:

By wearing helmets, cyclists are at best only marginally reducing their chances of being fatally or seriously injured in a collision with a motor vehicle which is the predominant cause of these injuries. Even the most expensive ones provide little protection in these circumstances. Moreover, the argument in favour of helmets would have validity if there were proof that behaviour does not change in response to perceived risk. But there is no such proof. Safety devices encourage higher levels of risk-taking. As a result, cyclists are likely to ride less cautiously when wearing a helmet owing to their feeling of increased security. After all, the message of the advocates of helmet wearing is that such a practice will protect the cyclist’s head adequately in the event of any accident, not just a minor one when cyclists are hit by very slow-moving vehicles or fall off and hit their heads on the ground. Cyclists may be less likely to have an accident if they are not wearing a helmet, and are therefore riding with greater care owing to an enhanced sense of their vulnerability. 

Furthermore, people are discouraged from cycling if their perception is heightened that it is a ‘dangerous’ form of travel and that it is only safe to do so if a helmet is worn. The result of this is that the considerable latent demand for cycling – an ideal mode for the majority of the population for most of their journeys – continues to be suppressed. As cycling is also a convenient and routine way of maintaining fitness, a significant route to public health is prejudiced. 

There remain then three questions to be answered. First, should helmet wearing be made mandatory? The report on which this paper is based has revealed no case for such a law. In addition to the absence of proof that helmet wearing reduces the risk of head injury, such a law would represent an infringement of civil rights. Moreover, where it has been introduced, it has led to a significant reduction in cycling

The second question to address is whether, whilst not making it mandatory, cyclists should nevertheless be encouraged to wear helmets -in effect, obliged to do so by ‘moral’ persuasion rather than by law. However, other than concern on the civil rights issue, the approach to helmet wearing by this means rather than by coercion through legislation would appear to be equally invalid. 

This then leads to the third question concerned with alternative and effective ways of reducing the risk of accidents, and therefore of head injury, among cyclists. The primary means of reducing serious head injury among cyclists is to create an environment in which accidents are less likely to occur. Such a strategy based on tackling the source of accidents in which cyclists are involved has far greater scope for reducing head injuries than the questionable benefits of promoting helmet wearing among cyclists.

 

The 2005 report form the National Children Bureau focused on cycling safety, not just cycling helmets.  The section on bicycle helmets, written by an experienced cyclist who regularly wears a helmet, concludes:

“The conclusion from the arguments outlined above is that the case for cycle helmets is far from, sound. The strong claims of injury reduction made by helmet proponents have not been borne out for fatalities (which this paper argues is the most methodologically sound test of effectiveness) in real-life settings with large populations. …

the benefits of helmets need further investigation before even a policy supporting promotion can be unequivocally supported. ….

The cycle helmet debate shows the dramatic power of real life events in shaping our understanding of causality. Tragedies happen; child cyclists are killed or left disabled for life; and we cannot let go of the belief that something
could and should have been done to stop that particular event from happening – especially when that something is so simple as wearing a helmet. We find it hard to accept that the helmet may have made no difference. We find it harder to accept that encouraging or forcing children to wear helmets might also encourage them to ride in a more dangerous way and paradoxically to increase the risk that they will suffer an accident. And we find it much harder to accept that compulsory helmet use might put children off cycling altogether, leaving them less physically active, and – many years later – more likely to die of heart disease. Think of all the uncertainty behind that line of argument, compared with the seeming rock-solid conviction that a helmet could have saved that particular child’s life, at that particular time. And of course the fact that we are talking about children, who have a claim on our protection and who are still getting to grips with the world, makes it so much more difficult to accept the limitations on our ability to prevent them coming to harm. We cannot ignore the human suffering, pain and loss that lie behind the research and statistics. But our response to it demands reflection and perspective as well as sympathy and conviction.

 

In 2011, the Transport and Health Study group, an independent British society of public health and transport practitioners and researchers, released Health on the Move 2, a book aimed to be “a clear and comprehensive account of what would constitute a healthy transport system.”  The section on bicycle helmets evidence, approaches the topic of the helmet law from a health perspective.  It includes a summary of the research on the helmet law.

The report is written in a neutral tone, yet you can sense the frustration of the researchers here: The failure of mass helmet use to affect serious head injuries, be it in falls or collisions, has been ignored by the medical world, by civil servants, by the media, and by cyclists themselves. A collective willingness to believe appears to explain why the population-level studies are so little appreciated.”

The report conclusions include:

The disconnect between received wisdom and the facts is stark.

The facts are:

2. There is however a disturbing discrepancy between engineering or clinical evidence of the effectiveness of helmet wearing (which suggest them to be effective) and population studies (which suggest that they are not).
3. Plausible explanations of this discrepancy include cyclists taking greater risks because they think their helmet makes them safe or drivers taking less care of helmeted cyclists because they see them as less vulnerable. A single study has examined this but its findings supports the latter of these.

5. It is now well established that legislation mandating cycle helmet use causes a reduction in the levels of cycling and thereby does more harm than good.
6 It is unclear whether this is because many people find cycle helmets troublesome, because many people find them unfashionable and odd or because people consider the mandation of helmet use as evidence that cycling is dangerous.”

Helmet believers doubtful after increase in injuries

A helmet advocate researched Canadian provinces with a bicycle helmet law. What he found shocked him. Provinces with a helmet law experienced relatively more injuries.

“A study that compared six-year periods on either side of the helmet laws in the four provinces that have them calculated a reduction in fatalities of 37 per cent and a reduction in cycling of 20.5 per cent, for a net reduction in fatalities of 20.4 per cent. In provinces without helmet laws, there was a reduction of 29.5 per cent.

Other Canadian studies reports a decline in cycling and an increase in the rate of injury for children.

“Compared to adults who were not required to wear helmets, children’s cycling (<13 years) fell by 59%, with a 41% reduction for teenagers aged 13-17 …

The observed post-law number of injuries – 1676 per year – is 2.37 times higher than would have been expected for the amount of cycling.  In contrast, the safety of adult cyclists (who were not affected by the law) improved.

Thus, far from improving safety for children and teenagers, the risk of injury seems to have increased after Alberta introduced its helmet law.  Similar calculations (Tables 2 & 3), show increases in the risk of head and non-head injuries requiring ER treatment for both children and teenagers, as well as increased risk of head injuries for children, and non-head injuries for children and teenagers admitted to hospital.  In contrast, risks for adults generally decreased.”

Similar result in the US, as reported by the New York Times:

the rate of head injuries per active cyclist has increased 51 percent just as bicycle helmets have become widespread. …

the increased use of bike helmets may have had an unintended consequence: riders may feel an inflated sense of security and take more risks. …

The helmet he was wearing did not protect his neck; he was paralyzed from the neck down. …

”It didn’t cross my mind that this could happen,” said Philip, now 17.

”I definitely felt safe. I wouldn’t do something like that without a helmet.” “

A recent study in New Zealand reports a similar result:

“The New Zealand Medical Journal research found a 51 per cent drop in the average hours cycled per person from the 1989-90 period when compared to 2006-09. …

Comparing the ratio of cyclist to pedestrian injuries from 1988-91 to 2003-07 showed cyclists’  injuries more than doubled compared with pedestrians “

Similar result in Australia. Cycling decreased by 40%, the risk of injury tripled.

The most surprising outcome of bicycle helmet laws is an increase in the rate of head injuries.

Do helmets really protect racing cyclists?

Most sport cyclists wear a helmet as it is “obvious” it protects.  Does it?

Since mandatory helmets, deaths of professional cyclists while racing have doubled:

“The helmet rule for professional cyclists was brought by the UCI in 2003 following the death of Andrei Kivlev during the Paris-Nice race.

Since then deaths of professional cyclists while racing have doubled, so where is the protection that helmets are supposed to give a rider?

It seems to me that there is too much emphasis on the part of manufacturers in designing something that looks cool rather than do what it is supposed to do, and that is protect a rider in the event he or she should hit their head.”

Dr Carwyn Hooper from St George’s University in London reports:

“Looking at evidence, it does not matter if people are wearing a helmet or not, any serious accident on a bike is likely to kill them,”

Polystyrene helmets are designed to mitigate falls below 20 km/h. Professional cyclists travel at speeds of 40 to 50 km/h, increasing to 80 km/h in downhill sections.

Why is it deemed essential that racing cyclists wear devices that are not designed to protect them? The Union Cycliste Internationale, the sport’s ruling body, first tried to impose helmets in 1991, but failed. It finally imposed them in 2003, using a tragic accident as an excuse. This opened up a new stream of sponsorship income for professional cycling.

Bicycle helmets:

  1. increase the risk of accidents
  2. increase the risk of neck injury
  3. increase the risk of brain injury

Does the protection compensates for these risks?

Misguided study affected by confirmation bias

Abstract

Some studies attempt to “prove” that helmets reduce injuries rather than genuinely attempt to understand their actual protective effects. Bold claims are made. Yet the data does not support them. While such studies are well meaning, their misleading claims can push people towards counterproductive solutions. 

 

One example is this study, that states it is designed to

demonstrate the safety benefits of helmet use“.

It seems that the researchers assume that helmets are effective and attempt to prove it. This lacks scientific discipline. It seems like an attempt to prove a preconception. With such an approach, there is a strong tendency for confirmation bias. People seek to confirm their existing beliefs.

The bias is revealed in this newspaper article:

“But the percentage of injured cyclists who needed treatment for serious head injuries dropped from 10.3 per cent in 2005 to 2.5 per cent last year.”That could be put that down to helmet use,”  “

What is odd about this conclusion is that helmet usage in the area was lower in 2009 than in 2005. Lower helmet usage was correlated with lower head injuries. How can lower head injuries can be credited to helmets when fewer people were wearing helmets?

The study report indicates unhelmeted cyclists had an Injury Severity Score (ISS) of 7, much higher than helmeted cyclists with an ISS of 4. ISS reflect all injuries, not just head injuries (25% of injuries). A higher ISS indicates more severe accidents. Differences in head injuries cannot be fully attributed to helmets. Accident severity affected the outcome. This is a common error is statistics, not checking for confounding variables. This renders the study conclusion invalid:

“The study confirmed the utility of helmet use in preventing serious head injury after cycling accidents.  This was the only factor in this study to influence the severity of injury.”

How can the researchers have missed accident severity as a relevant factor?

This is confirmation bias. Researchers look for data that matches their existing beliefs. They tend to miss data that does not confirm their belief.

Despite the lack of scientific discipline, the bias of the researchers, and the flaws in the study, the government claims that this “confirms” that helmets are effective.

This is typical of studies attempting to “prove” the effectiveness of helmets. While such studies are well meaning, they can mislead people towards false “solutions” to cycling safety, while more effective measures are neglected.

First Ad to repeal the helmet law

An Australian cyclist has made the first ad to repeal the bicycle helmet law, after being inspired by the level of utility cycling among all ages in a trip in Italy.   The ad was even available in the mainstream media, not often that australian media shows an ad for free!

The ad invites people to take action through HelmetFreedom.

It has attracted a bit of attention from the media, here and here.

The QLD government felt the need to reply to it , quoting misleading claims made by a “study” it commissioned, even though it had been rebutted here and here.