Peer review example: “Effectiveness of the BNT162b2 vaccine among children”

Having written hundreds of requested peer reviews, I am offering my unsolicited one.

“The Hill” has recently published one reporter’s analysis of the pre-print article “Effectiveness of the BNT162b2 vaccine among children 5-11 and 12-17 years in New York after the Emergence of the Omicron Variant”.  The title of the analysis by reporter Peter Sullivan is “Pfizer vaccine significantly less effective in kids 5-11: study”. 

Steve Bannon has asked that I come on the “War Room Pandemic” broadcast during the AM session today (01 March 2022) and discuss the article (as well as the press response).  So, to prepare for this broadcast session (the slang used is “hit” – I have been learning so much about broadcast media!), I decided to just approach this as a standard academic peer review. 

The goal with this essay is to give you a peek into what an independent peer review of an academic manuscript might look like.  I go through a similar process when I review grants and contracts, but in those cases I have the benefit of a “biosketch” for the authors, as well as a detailed budget, clear statement of purpose, detailed description of methods etc.

When reviewing, I first go to the abstract. Frankly, if that is not concise and well written, it influences my approach to the entire paper. Let’s see what we have here.  My comments in italics.

Importance: There is limited evidence on the effectiveness of the BNT162b2 vaccine for children, particularly those 5-11 years and after the Omicron variant’s emergence.

Nicely stated, frames the context quite well.

Objective: To estimate BNT162b2 vaccine effectiveness against COVID cases and hospitalizations among children 5-11 years and 12-17 years during December, 2021 and January, 2022.

OK, there was still some Delta circulating during that time frame, but this is likely to mostly be Omicron.  Need to look and see whether the authors document the strain(s) infecting the sampled population analyzed, or are they just inferring based on broader statewide trends. Cases and hospitalizations – that basically means inpatients and outpatients.

Design: Analyses of cohorts constructed from linked statewide immunization, laboratory testing, and hospitalization databases.

Aha! This is a case cohort study, not a prospective study, let alone a randomized study.  Not the pinnacle of scientific evidence.  Generally useful for identifying trends.  Often the best one can do in epidemiology, and often this design can be quite predictive.  The main problems with case cohort studies is selection bias and unidentified/uncorrected confounding variables.

Setting/Participants: New York State children 5-17 years.

Well defined and constrained. A specific subgroup for analysis, less prone to unidentified/uncorrected confounding variables relative to a broader analysis window.

Main outcomes/measures: New laboratory-confirmed COVID-19 cases and hospitalizations.

Ok, this approach introduces some selection bias, but on the positive side it reduces confounding due to reporting bias. Need to look carefully at how they define “laboratory-confirmed”.

Comparisons were made using the incidence rate ratio (IRR), comparing outcomes by vaccination status, and estimated vaccine effectiveness (VE: 1-[1/IRR]).

Again, well defined, appropriate, clearly stated. So far I am really liking this paper.

Results: From December 13, 2021 to January 30, 2022, among 852,384 fully-vaccinated children 12-17 years and 365,502 children 5-11 years, VE against cases declined from 66% (95% CI: 64%, 67%) to 51% (95% CI: 48%, 54%) for those 12-17 years and from 68% (95% CI: 63%, 72%) to 12% (95% CI: 6%, 16%) for those 5-11 years. During the January 24-30 week, VE for children 11 years was 11% (95%CI -3%, 23%) and for those age 12 was 67% (95% CI: 62%, 71%). VE against hospitalization declined changed from 85% (95% CI: 63%, 95%) to 73% (95% CI: 53%, 87%) for children 12-17 years, and from 100% (95% CI: -189%, 100%) to 48% (95% CI: -12%, 75%) for those 5-11 years. Among children newly fully-vaccinated December 13, 2021 to January 2, 2022, VE against cases within two weeks of full vaccination for children 12-17 years was 76% (95% CI: 71%, 81%) and by 28-34 days it was 56% (95% CI: 43%, 63%). For children 5-11, VE against cases declined from 65% (95% CI: 62%, 68%) to 12% (95% CI: 8%, 16%) by 28-34 days.

Very nicely stated.  CI = confidence interval, which is the statistical range (in this case they have used 95%) surrounding the calculated vaccine effectiveness value.

Conclusions and Relevance: In the Omicron era, the effectiveness against cases of BNT162b2 declined rapidly for children, particularly those 5-11 years. However, vaccination of children 5-11 years was protective against severe disease and is recommended. These results highlight the potential need to study alternative vaccine dosing for children and the continued importance layered protections, including mask wearing, to prevent infection and transmission.

I have problems with this. The first sentence is the conclusion. The rest is editorial and not supported based on the preceding sections of the abstract. The results say nothing about mask wearing to prevent infection and transmission in this cohort. How does this study assess disease severity? A disease severity endpoint was certainly not included in the experimental design as outlined above. Why are these authors stating an opinion about public health recommendations?  This is neither justified nor supported. This “Relevance” statement is basically yet more propaganda which the authors appear to have inserted because academic journal editors require this type of genuflection to the approved narrative.  If I were the reviewer, I would call BS on this.

At this point, I have lost confidence in the objectivity of the article and the authors, who have clearly demonstrated a lack of objectivity. They have written the relevance section in a way that demonstrates that they have an agenda.  Now let’s look at the authors, their affiliation, and potential conflicts of interest.

Vajeera Dorabawila, PhD, Dina Hoefer, PhD, Ursula E. Bauer, PhD, Mary T. Bassett, MD, Emily Lutterloh, MD, Eli S. Rosenberg, PhD

New York State Department of Health, Albany, New York

University at Albany School of Public Health, State University of New York, Rensselaer, New York

No conflict of interest statement.  In the world of modern academic manuscript submissions, this is absolutely not acceptable. First big red flag, particularly in light of the clear bias demonstrated in the “relevance” section. 

At this point, I am rapidly losing confidence in the integrity of this manuscript. They are basically PhD and MD public health bureaucrat/officers of the State of New York. The absence of a conflict of interest statement is particularly notable. Usually the first and last authors are the most important.  So now I turn to PubMed and look up the publication record of these two.

Vajeera Dorabawila, PhD. Eight prior cited publications. No prior first author. Second red flag.  Prior MMWR (ergo CDC non-peer reviewed) publication. This person is not a thought leader, and is very much a part of the standard public health bureaucracy. COVID-19 has been the main focus of her “academic” manuscript work product.  Little to no prior experience.

Eli S. Rosenberg, PhD.  150 manuscripts cited on PubMed. Checking on Google Scholar. Bingo.  This is the big gun behind this manuscript. Title: Deputy Director for Science, Office of Public Health, New York State Department of Health as well as Associate Professor, Epidemiology & Biostatistics at University of Albany. Trained at Emory. Emory public health is basically a satellite of the CDC, or the other way around, depending on how you look at it.  Here is his webpage.  So, from this I conclude that the leadership here is well trained, experienced, but very much bought into the CDC narrative.  Which we now know is basically propaganda grounded in scientific fraud (in my opinion).  So, there is that.  Good to know.

With that background and context in mind, lets dive into the body of the paper, looking for whether or not those “red flags” and other concerns have merit.

First question – are there aspects of the experimental design not clearly covered in the abstract?

Well, yes.  Now we get to the bottom of what is actually being compared here. This was not well described in the abstract. Vaccine effectiveness comparisons between the 12-17 year old cohort (dosed with 30 micrograms of the EUA Pfizer/BioNTech product- the standard adult dose) x2 relative to the 5- 11 cohort who are dosed with EUA Pfizer/BioNTech product at 10 micrograms (1/3 of the adult dose) x2.  So, these children have not received a third shot. Only “fully vaccinated” children )defined as status post 2 doses + 14 days) were analyzed.

If we look at the reported data analyses (see Table 1), what immediately first jumps out to me is that we do not have the usual summary table of enrollment characteristics.  This is not consistent with accepted practice when reporting a clinical study – particularly a retrospective study like this. The consequence is that the reader has no idea about potential imbalances in enrollment between the groups other than that which can be inferred from Table 1.  This would be reason enough for me to reject this paper at this point, or at a minimum to require a major revision. This deficiency might be more allowable if we did not have the highly experienced last author. But Dr. Rosenberg knows better. A third red flag.  At this point, I would probably apply the three strikes rule. This is looking more and more like propaganda and less and less like a rigorous study.

What we can see in Table one is a major imbalance in enrollment between vaccinated and unvaccinated children. This is yet another warning sign of potential selection bias.  Fourth red flag. There are likely to be socioeconiomic differences between these two groups. At this point, I am increasingly thinking that this report should be pulled from the pre-print server. It fails to meet even minimal standards.

Furthermore, this study is not from a single database, but rather aggregated data from three databases.  Aggregating data from multiple databases often can lead to analysis artifacts. This raises another question – what is the balance between the two analysis groups in these various databases?

Note that this study is limited to analysis of children newly fully-vaccinated in the 3 weeks from December 13, 2021 to January 2, 2022.  Another form of selection bias. The title and conclusion is therefore misleading. This is not a study of “Effectiveness of the BNT162b2 vaccine among children 5-11 and 12-17 years in New York after the Emergence of the Omicron Variant” but rather it is a grossly imbalanced study lacking matching between cohorts of Effectiveness of the BNT162b2 vaccine among recently fully vaccinated children 5-11 and 12-17 years in New York after the Emergence of the Omicron Variant.

Figure 2 is not adequately labeled and the figure legend is inadequate.  Figures should have legends which stand alone and completely describe what is being shown.

Second question – Are there any data concerning masking in this cohort of children?

To be blunt, no.  Any comments regarding mask use of lack thereof are irrelevant and unsupported, and have no place in this manuscript.

Third question- Does this study discriminate between whether the children were infected with Omicron or Delta?

Again, no, not at all. Another major flaw. The title of the manuscript is technically acceptable, despite this flaw, as it states “after the Emergence of the Omicron Variant”, but it is somewhat  misleading.

Fourth question- Are there data allowing comparisons which would support conclusions concerning severe disease?

In my opinion, the conclusions regarding relative vaccine effectiveness for severe disease are preliminary estimates at best.  The study results are highly likely to be biased by confounding between the vaccinated groups.  Lack of any table summarizing the two groups further compounds this concern. At this point, as a reviewer I would strongly recommend rejection of this manuscript.

Finally, this last conclusion statement is pure editorial opinion/propaganda. This conclusion is not supported by the data.

 “Given rapid loss of protection against infections, these results highlight the continued importance of layered protections, including mask wearing, for children to prevent infection and transmission.”

---

Despite the above caveats, review of Table 1 strongly suggests a profound lack of effectiveness of the Pfizer/BioNTech mRNA COVID-19 vaccine in children.

Once again, in my professional opinion, there is no justification for mandated use of this product for children, and no justification for use at all in healthy children.

---

And that is the way it is done.  Reject for publication.

Which illustrates another key point.  Why are we relying on reporters to interpret scientific articles?  They lack the necessary training and expertise.  The lay  press, including “The Hill”, have clearly been the primary purveyors and amplifiers of mis- and disinformation throughout this outbreak.  These “journalists” are often of the modern advocacy journalism school (that means propagandists in plain speak), and are not even journalists in the old school (“fair and balanced”) sense, and definitely are not scientists or physicians.

The press and “advocacy journalism” reporters need to get back in their lane and leave scientific and medical interpretation to experienced professionals.

And stop trying to spin that which they do not even comprehend.