How much should we trust our data? In the face of unusual data, when do we toss it out or go after a groundbreaking theory?
This is a permanent question in the quest for knowledge. There is no simple answer, but it helps to have guidelines and ways to check for mistakes.
I see this in a very simple way when my students solve problems in the classroom. they tend to trust that the number after the equal sign is the 'truth' because it is at the end of the equation, or that the value read from a sensor is correct because that is what sensor do. I ask them to do two simple checks to verify their numbers:
Use units in all of your calculations and make sure they all match up all the way to the end. Then check that the unit you got corresponds with the quantity you were looking for. This means that if after doing all the math you get a unit of grams when you were looking for length, you should know you made a mistake.
Make sure the quantity you got makes sense in our view of the world.
The first check is pretty simple and unquestionable. If you start making an orange juice and end up having grapefruit juice you know you made a mistake, as simple as that.
The second one works well in the classroom when problems are bound by our lack of imagination. If a student is doing a mass calculation for a real drivable car on a given problem, they should realize they made a mistake when they get a mass of 12 grams.
But it is not that easy to distinguish flaw data from real unknown features when doing real science. We are lucky to have aboard the most amazing experts. I have heard people on board tell Kristin and Mary that they are legends on what they do, and from what I have seen them do, I believe that to be the case.
Kristin and Mary represent the data command control. Everybody in the ship sends their analysis results for them to compile the data base. They gaze at different type of plots looking for suspicious numbers. They did an amazing detective investigation to solve a confusing riddle. Even Kristin, with a smile in her face, recognized that this was a challenging one. She spent a whole day looking into the data from a single cast (rosette deployment). Here is how they solved the problem.
The rosette obtains temperature, salinity, pressure and dissolved oxygen data from the CTD at a rate of 24 measurements per second (that is a lot of data!). The CTD has two temperature and salinity sensors that help us know if one is not working properly. We also analyze the salinity of each bottle. We check for an agreement in salinity between the CTD and the bottle data up to 0.002 ppt (parts per thousand). We can detect when the CTDs drift and calibrate them as we go.
The first mistake that they found on a cast was easy to spot. The bottom bottle had a salinity of 1 ppt lower than the CTD at that depth, and 1 ppt lower than the nearby bottles. How could a bottle have so much fresher water? Further research showed that the nitrates and silicates were very low for a bottom bottle. Actually, they looked like the values at the surface; the same with the salts. Here is the spread sheet used by Kristin to figure out the mystery.
Kristin used this spreadsheet to make annotations as she figures out the mistake.
The first column on the left of the spreadsheet is the sample number with a 1 for cast number at the station and then the bottle number. The following column is pressure from the CTD.The third column is depth on meters, the fourth is temperature, then they are salinity the bottle, salinity from the CTD, the difference in salinity between bottle and CTD multiplied by 1000, dissolved oxygen from the bottle, dissolved oxygen from a sensor on the rosette, their difference, nitrates, phosphates, nitrites, silicates.
The analysis of the bottom bottle did not match the values from deep waters, but it matched the surface values. This suggests that the bottle closed at the surface, not at the bottom. How can this be?
Let us go back to how the bottles work. In the journal entry of February 25 we learned that the bottles are attached to a circular frame, and that they are kept open by a monofilament lanyard that attaches to tiny hooks on the carousel. You can see in the picture below how close the lanyards are when attaching to the tiny hooks
Rosette carousel. Look at how closely spaced are the lanyards in the carousel.
Bottle 1 should have tripped at the bottom and bottle 36 at the surface. Remember the circular shape of the rosette, this means that bottle 1 is next to bottle 36. I am sure you can figure out what happened that made bottle 1 trip at the surface. Write me through the 'Ask the Team' if you have not figured it out and I will give you the explanation.
This means that bottle 2 closed at the bottom, and bottle 3 closed where bottle 2 should have closed. From then on, all of the bottles are shifted one spot down. Problem solved, right? Not quite.
You can see that Kristin wrote in pencil the new corresponding bottle number for each pressure until 10 (lower left corner of the spread sheet). She stopped there because she smelled another mistake…
This one is harder to spot. Short story, she realized that two bottles had tripped at the same time because the person in charge of setting the bottles placed the lanyards of two bottles on the same carousel hook. Here is a graph of how data looks like when two bottles close at the same time. The vertical axis is depth in meters and the x axis has different values according to the variable.
The graph shows that bottles 5 and 4 were tripped at the same time.
In this other cast, bottles 5 and 4 tripped together. There is a 2 in front of the sample number because the trace metal rosette was first on that station. You can see that the oxygen (red line), silicates (cyan) and the phosphates (light green) have exactly the same value for both bottles. The problem for Kristin is that somehow the nitrates (blue) are not equal in both bottles, a fact that confused her at first. When she looked at more data she was convinced that the bottles had tripped together when she looked at the other bottles as well. They solved both riddles and made sure that the people who get the bottles ready really understood they needed to be more careful. We have not seen lanyard problems since.
It should be noted that Kristin and Mary summon all brains around to confirm their ideas. And even when a decision is made that there was a human error, all data is reported. As Kristin said, the good, bad and ugly goes into the records for others to make their own decision about what values they want to use.
Overall it is not simple to find an explanation for unexpected data values. Sometimes we presume values that show up differently because they do represent interesting phenomena. Jim discovered the Arctic Ocean Deep Water outflow on a cruise while watching the CTD values on the screen as the instrument reached the bottom. The data showed high salinity values that were unexpected at that depth. They did not find an error in the procedures to account for those values, so he proposed the outflow of the Arctic Ocean Deep Water. The existence of this salty bottom water has subsequently been confirmed.
Jim says that we do not toss suspicious data just because it looks different. We need to find an explanation that accounts for a mistake or for a novel idea that explains it. That is science at its core!
There is another oceanographic example of how data that looks wrong ends up supporting new models. In the old days, before the silicon revolution, oceanographic instruments were all mechanical. We did not have CTDs spewing data like we do now, and all of the salinity measurements came from bottles. For decades people collected data from the Atlantic Ocean, and every now and then there were salinity measurements from bottles around 1000 m that were out of place. Records show people labeled them as bad data because they thought there was something wrong with either the water collection or its analysis. Then Larry Armi, from Scripps Institution of Oceanography, proposed a new idea: the data shows the presence of subsurface eddies from the Mediterranean outflow. These eddies are spinning blobs of very salty water that leave the Mediterranean at about 1000 m. They remain distinct because of their spin, just like a top does not fall when it is spinning. They are not large enough to be fully resolved with spotty bottle data. Modern CTD data has confirmed the existence of these Mediterranean eddies. Oceanographers now cal these features 'Meddies'.
Science involves the delicate balance between approaching a problem with an open mind and lots of creativity for solving it, and having a good sense of what can be expected. It was exciting to see masters at work in the art of discriminating data.