Tuesday, 7 February 2017

Data interpretation, the failure missed

HIGHLIGHT

  • You have to read the data, not someone's interpretation of the data, as that may be wrong.
  • Base your assumptions on absolute numbers not percentages
  • Serious misinterpretation of data can be made, when basing ideas on percentage changes

On Thursday/Friday we made the case for the importance of B cells.

http://multiple-sclerosis-research.blogspot.com/2017/02/ms-is-b-cell-disease-and-memory-b-cells.html

Yesterday, we made the case that the data from clinical trials argued against a role for T cells. 

http://multiple-sclerosis-research.blogspot.com/2017/02/is-ms-t-cell-mediated-disease.html

MrT was quiet. 
Maybe MrT knows better than to say something:-). 
Maybe he/she has changed his/her world view.  

Have You?

There was not too much debate

This says you don't either understand what I am on about, 
you don't care or perhaps 
you don't know how to read the data ....................otherwise you may have called me on it, because in some cases I was talking rubbish.

What's new...I hear you say:-(

You would think, I would get called out, 
there are people who accept this thing without question. 

Sadly it is often made by someone called a referee:-(.

However, there are always two sides to a story and I only gave you one yesterday to make a point. 


So forgive me if you feel you were led astray

There are a large number of papers reporting an increase in the levels of memory T cells after alemtuzumab and fingolimod, as I showed you. 

However, this view is complete and utter non-sense of course, 

So the calls to ProfG by Pharma were wasted (white lie:-), alemtuzumab and fingolimod both deplete memory T cells.

So what did I show? 


I showed that the proportion of CD4, CD45RO goes up.

To make an example let's say CD4CD45RO increase from 10% to 40%, so it is a thirty percent increase right?

Wrong! It could be a number of things. 


The technique that people, typically immunologists, use to monitor cell numbers is a flow cytometer (see above) 


This detects markers say CD4 for T cells and a marker for CD45RO (memory cells) and it gives you a percentage as a read-out. So it can say of the CD4 population, x% also express CD45RO.  The flow cytometer counts say 10,000 cells per sample so in the population there were 40,000 CD4, CD45RO to be 40% or 10,000 to be 10%. This also means in the negative populations there are 90,000 or 60,000 cells. 


But is this increase because there are 30,000 more CD4, CDRO positive cells or because there are 30,000 less CD4, CD45RO negative cells? 

So in addition to knowing the percentage of cells, you need to know the absolute cell number in any given volume, so you can work out the absolute number of cells in the sample. 

This aspect is lacking in many of the publications.
This makes it very difficult to interpret.

So we could have 


10 positive + 90 negative = 10%,  1000 positive + 9000-ve = 10%  
 40 positive + 60 negative = 40%, 4000positive + 6000-ve = 40%

So in normal health  it could be that there are 1000 positive cells per mL 
and 9000 negative cells in 1 mL = 10%, but after depletion there are 40 cells that are positive cells per  mL and 60 that are negative so 40% = 30% increase in the proportion of cells, but if we say that within the mL the number of positive cells have gone from 1000 cells per mL going down to 40 cells/mL. 

Therefore in absolute numbers the numbers of cells have gone from 1000 down to 40 so actually a 96% depletion not a 30% increase. 


Do you get this...many, many scientists don't:-(. 


However, this has led to a problem because you are getting a misleading interpretation.

Yesterday, I showed you the figure from the paper indicating an increase in the effector memory (TEM) population, arguing it can't be T cells.

In the abstract it said "In MS patients, the percentages of central memory T (CCR7+CD45RO+) cells (TCM) and naïve T (CCR7+CD45RO-) cells decreased significantly, while those of effector memory T (CCR7-CD45RA-) ......increased in both CD4+T and CD8+T cells from 2 weeks to 12 months during fingolimod therapy". So authors implie the cells go up but....

Look in the TEM In the data set the first two columns are controls the next six are time after treatment so you can see the line = median halfway point goes up.

Tucked away in the supplementary data, not seen by many, is the truth. 

Of course effector memory T cells are depleted you just have to look at the absolute number of cells. The effector memory cells are depleted, but as a proportion not as much as naive and central memory.
Look at the TEM group. In the data set the first two columns are controls the next six are time after treatmentso you can see the line = median halfway point goes down

Shocking yes ......but this type of analysis is not uncommon and I am sorry to say I believe that this type of analysis has led to failure to appreciate the effects and side effects of MS drugs, and probably put people to un-necessary risk (posts being written to explain this). 

However, to go back to the point. So redoing the table of moderate/highly effective drugs based on absolute numbers you still can't argue for a direct action on depleting T cells of active drugs.
_____________________________
                    Memory T  Memory B 
_____________________________
Alemtuzumb    YES           YES
Fingolimod      YES           YES
CD20               NO             YES
____________________________

T cellers still struggle and more bad news to follow

But the effects of the anti-CD20 can be indirect and block T cell function, leaving them to fight another day. 

Remember to be a lemming without questioning, can mean you end up over the cliff :-).

http://multiple-sclerosis-research.blogspot.com/2012/02/education-same-old-same-old.html

13 comments:

  1. Too long. Rambling. Incomprehensible. No structure. Very disappointing. Need to use next week's half term to refocus. Think through what message you are trying to convey.

    ReplyDelete
    Replies
    1. Soory Teach...you simply have to read this one and not skim read, then re-read because this is the important piece on the curriculum

      P.S. We will revise this one, just for you, because the exam question has been set "Can you understand the problems of treatment and why there is an un-appreciated danger".

      Delete
  2. But MD surely this is why papers have to be reviewed prior to publication? Referees should be looking out for this kind of thing? Then busy doctors can read journals without a toothcombe. I'm clearly naive ;-)

    ReplyDelete
    Replies
    1. The review is only as good as the job the reviwer does and whether it is constructive of not or whether they want to be arrogant or whether they want to get the paper rejected....How you just mention lack of novelty in many cases thats enough.

      The other point is ifthe reviewers read the post...For our B cell paper
      for example the referee said "it would be good to show for clinicans the different types of B cell and the surface markers that they express".

      Obviously the reviewer was a basic scientist, so no wonder they held on dearly to their T cell dream.

      But to be polite we answered
      " We have included figure 3 to show a simplified view of B cell development and the markers they express.

      This implies to the editor who often does not read the papers (just the reviewers comments) that we have done something to address the reviewers concerns.

      However we had to change the first response which took on board DrKs comment "Should have gone to Specsavers". Figure 3 was already in the original manuscript and if the referee had read and understood the paper half the comments would not be made. This is where the frustration arrives.

      Delete
  3. lol if i was a stats guru i would not be turning to this blog, another blog/group run by an ms researcher, my father (an engineer lol) and my partner's neuro to try and decipher stats. still, i am guessing this is not aimed at me.

    who is the mr t reference to?

    "Shocking yes ......but this type of analysis is not uncommon and I am sorry to say I believe that this type of analysis has led to failure to appreciate the effects and side effects of MS drugs, and probably put people to un-necessary risk (posts being written to explain this). "

    I seriously want to be shocked, but I don't follow enough to be even get close... can you explain using a simple metaphor, please :)

    ReplyDelete
    Replies
    1. It referes to the use and abuse of a machine, it is not about statistics it is about data analysis.

      Is it aimed at you? If you read the science papers then yes...if not then you accept the conlusions made using the machine, because it is about understanding data.

      MrT is a blogger who thinks the B cell idea is clap trap but there has been no killer reference offered to argue why the B cell hypothesis is no good.

      Simple metaphor....read the post slowly

      Read the Highlights it explains it all.

      If you are struggling now I wonder what happens next, when the real data is shown.

      Delete
    2. What a strange reply to my comment. Perhaps you are feeling sensitive today.
      Data analysis relies on knowledge of statistics – amongst other things. I read science papers too much for my liking – and I read them because I feel the society and the individual professionals have let me down: I don’t trust anyone anymore, least of all doctors and scientists. I do have a day job, which relies on words, not data analysis. If it helps, I can argue that the sky is purple if I need to and often win the argument.

      You have to read the data, not someone's interpretation of the data, as that may be wrong.
      • Base your assumptions on absolute numbers not percentages
      • Serious misinterpretation of data can be made, when basing ideas on percentage changes

      Great… but I still need a simple example to illustrate your point about absolute numbers. If you took the time to reply to me, surely it would have been just as easy to give me a simple example? I get the gist, but I want to understand and I need a simple example than what you have gone on to say (absolute numbers v percentages). I could google it, true, but I do have a day job (like you) and there are only 24 hrs in a day. You suggested you are a teacher in another comment to someone else ( “so perhaps this is a exercise of venting the frustration...whilst performing education as I see it.”)… an educator doesn’t really blame the audience when they don’t understand.. does he?

      Should I set a legal problem for you, then when you answer the problem with the help of google searches without understanding the intricacies involved in the problem and the complexity of the law, should I suggest you’re not trying hard enough and are a slave to the machine that is our society?

      What will happen when the real data is released? I will ask you questions and hope you are in a better mood to answer them, I will google the basics and write notes as I go, I will ask the other scientist I know, my father and my partner’s doctors and compare the answers and if I’m still not understanding or dissatisfied with either my understanding or the inconsistencies, then I’ll find someone else to help me. The same thing that’s been happening all along.

      What a strange suggestion – that laymen should understand data analysis in detail and fully – one I sincerely hope is based on some kind of frustration. Have you read the Bell Curve? Do you believe that some people are more intelligent than others – that some people have a bigger capacity for complex problem solving and analysis than others? Do you believe that the less smart ones deserve to suffer because they don’t understand the same things that you do? Where does that leave those people with MS who have cognitive details?

      You know, from all your rants against those who voted for the Brexit, I thought you were a leftie.

      Delete
    3. lol cognitive defects, not cognitive details.

      Delete
    4. I can argue the sky is purple and win....maybe thats why lawyers have much to answer for:-).

      The example you crave is coming, but first you need to understand the concept and then you wont need to ask questions:-)

      The real example was the two figures in the post.

      The inference that fingolimod increases memory T cells when it in fact is depleting them.

      That is all the data analysis you have to understand

      Is this clinically important probably not When the paper is accepted it will be posted but in the meantime. Some background information is needed to understand the work. This is one aspect of that.

      As for rants...kettle and black come to mind:-)

      The wrath coming my way, will not be from Australia I can assure you LoL

      Delete
    5. Dear Pot,

      As I’m sure you know I was picking on you to make a point, rather than because any of what I said about you is ‘true’. There was no wrath involved lol. My point was that your comment was unnecessarily mean towards people who don’t understand something you have written. I enjoy my rants, I think they’re well thought out and often have a ring of truth to them lol.

      Lawyers have much to answer for because they enjoy a position of power and because their lively-hoods depend on constructing such complex arguments on interpretation and analysis of words, apparently no layperson in the world can understand them. Does the law need to be as complex as it is in our respective countries? If I say no, I’m killing off my own livelihood lol.

      I got all of what you wrote in your comment about. I also get that 4 of 10 is 40%

      But I still don’t get this:


      The flow cytometer counts say 10,000 cells per sample so in the population there were 40,000 CD4, CD45RO to be 40% or 10,000 to be 10%.
      This also means in the negative populations there are 90,000 or 60,000 cells.

      and

      So we could have

      10 positive + 90 negative = 10%, 1000 positive + 9000-ve = 10%
      40 positive + 60 negative = 40%, 4000positive + 6000-ve = 40%

      So in normal health it could be that there are 1000 positive cells per mL and 9000 negative cells in 1 mL = 10%, but after depletion there are 40 cells that are positive cells per mL and 60 that are negative so 40% = 30% increase in the proportion of cells, but if we say that within the mL the number of positive cells have gone from 1000 cells per mL going down to 40 cells/mL.

      I just don’t get how the calculation is made.

      I don’t want to understand what you say I need to understand, I want to understand what I’m asking :)

      Thankfully yours,

      Kettle


      Delete
    6. I guess we all understand fractions so 1/4 is a quarter or a forth in some countries if I want to increase the proportion form 1/4 = 25% we can do it in two ways we can increase the numerator (the number on top) so we get 2/4 is now a half. Now see the numerator as positive cells expressing a marker e.g number of T cells However the other way to get the same thing is to decrease the denominator (number on the bottom) so we can have 1/2 which a half. So if you see the denominator as negative cells you can see that you can get an increase in the proportion of by either increasing the positive of decreasing the negative populations

      So if you have 10000 cells in 1 microlitre (a thousandth of a litre) of blood and the fraction of positive cells is 1/4 (1 positive to 3 negative = 25% so in the blood same you have 25% of 10,000 = 25 x 100 = 2500 cells. Now if I say the positive cells are good for health you have 2500 good cells in 1 microlitre of blood. Now if I give a drug and you have 2/4 cells are positive of good so now you have 50% of good cells in your machine that works on proportions. However the drug is actually killing cells so rather than having 10,000 cells in a microlitre of blood you only have 1000 cells per microlitre of blood if a half of them are positive or good cells then you only have 500 good cells so rather that the good cell number go up as implied if you only look at the proportion from 1/4 to 1/2 you think it goes up from 25% to 50% but in actual numbers you are going from 2500 cells down to 500 cells so there is a 80% decrease as 500/2500 is 20%. So less good cells
      So this would not be good news

      I hope you now understand.

      Now go to back to the example I showed you in the top graph there is an increase in effector memory cells as a proportion, but if you look at the bottom graph there is an absolute decrease in these cells, so fingolimod is depleting effector memory T cells from the blood.

      p.s. NDG has complained that you are bringing your rants about other things into her posts, and it means that people reading this will not understand that it is about a different post

      http://multiple-sclerosis-research.blogspot.com/2017/02/data-interpretation-failure-missed.html

      Delete
  4. MD, what you are doing is great! Looking at the data is an almost lost skill.

    ReplyDelete

Please note that all comments are moderated and any personal or marketing-related submissions will not be shown.