The challenge of 'cleaning' yield data from your fields

Saturday, August 8, 2015

Did your field really grow 350 bushels of corn? In the first of several articles, the authors look at tools you can use to check the reliability of the information you are getting about yield distribution – a key to precision agriculture

by MIKE DUNCAN and SARAH LEPP

The topic of cleaning yield data can be very tricky for some certified crop advisors and some farmers. The implication is that they've done something "wrong" and it needs to be fixed. The truth is that some of the required operations necessary to work a field properly just happen to be incompatible with collecting good data.

For example, the best data is collected by a combine going in a straight line, with a full header, and no speed variations at all. Inevitably, the combine is in a ditch or ravine at the end of such a run. The simple fact is that the combine has to slow and turn in the headlands, and that produces some really bad data. Fortunately, the bulk of the passes the combine makes in the field can be in a straight line and at constant velocity with a full header. (See Figure 1.)

So what should your yield data look like? The most useful mathematical tools that can be used to tell us about yield data is the histogram, or the probability distribution.

Histograms are formed from data, while probability distributions are proper mathematical formulas that apply to the data. So, we form a histogram from the yield data and infer what we can from it. A histogram is formed by looking at the highest and lowest yield value and splitting the interval into even "bins."  Figure 2 shows a histogram for 2010 corn data from the Culver-owned field in Simcoe County.

The average of the distribution is 254 bushels an acre, but the peak is at 267 bushels. The left or "low" tail of the distribution extends down to 12 bushels per acre, and the right or "high" tail extends to 407 bushels per acre. The collection pattern with the yield estimate points can be seen in Figure 1. The histogram is skewed to the right, or towards higher yields, which seems to be characteristic for multi-trait seeds. Single-trait seeds show a more symmetrical or bell-shaped curve.

The next obvious question is, what is real and what is not? Did this field grow 350 bushels an acre of corn? The first comparison that should be made is with yield histograms from previous years because, if this distribution is way off base, then comparing it to previous years should help decide. If it is off base, then there might be calibration problems.

What's frustrating is that, even if it's not off base with previous years, there might still be calibration problems, which can affect the average being measured and act to move the whole distribution up or down in value. The good news is that calibration problems preserve the relative highs and lows and so the yield pattern is preserved. However, they make it difficult to reconcile the amount of corn recorded by the monitor and the amount of corn finally sold.

What are some other sources of errors and their effects on the yield distribution? Speeding up or slowing down increases and decreases the flow of grain through the yield monitor, which artificially raises or lowers the yield estimates. The low yield measurements associated with slowing down, or a partially full header, or turning, will most likely affect the low tail of the distribution by producing low yield measurements. The higher yield estimates associated with the acceleration of the harvester tend to affect the high tail of the distribution.

There are also hiccups in the yield monitor that register either enormous or miniscule yields.  These things are a fact of life and the low values add to the low tail of the distribution while high values add to the high tail.

If we only pay attention to data collected when the combine's velocity is "constant," then we are probably looking at reliable data. Part of the data that the monitoring system collects includes the velocity at each point. Comparing velocity values point to point allows us to compute an acceleration at each point. As long as the acceleration is close to zero, we have good data unless the vehicle is turning.

Every farm business knows to the last kilogram how many tonnes were sold from a given field. Try matching that number against the total from the yield monitor and see what you get. I've done this for a few decades of totals from three or four fields and the differences are usually within about 30 per cent most of the time. Achieving an exact match is likely impossible due to moisture losses, spillage, measurement errors and a host of other factors. Getting to within 10 per cent consistently might be an achievable goal, and would give great confidence in using the yield maps.

A tremendous amount of research has revealed that predicting absolute yield is almost impossible; however, relative yield, even when there are monitor calibration errors, is unaffected. The stability pattern and knowing the relative differences between cells means that we can adjust relative amounts of fertilizers into each cell accordingly. This is the key to precision agriculture.

In the next article, we will talk about the actual "cleaning" of the distribution or how to remove the points that are likely to be errors. We will also answer the question as to whether this field grew 350 bushels an acre or not.

In future articles, we will deal with other forms of error that don't affect the distribution but make it almost impossible to practice precision ag. BF

Dr. Mike R. Duncan, PhD, is Natural Sciences and Engineering Research Council of Canada, Industrial Research Chair for Colleges in Precision Agriculture and Environmental Technologies, Niagara College Research and Innovation. Sarah Lepp is research associate.