DRONE VALIDATION
Measuring diameter manually vs autonomous
Introduction
We put our drone’s ripeness measuring application to the test at Kwekerij Baas. Currently, the ripeness of their Belgian Mums is measured by measuring their diameter from a top down view
Our goal is to measure this diameter with our drone as accurately as possible.
We performed a validation where we compared manual diameter measurements with the diameter measurements of our drone, to determine the accuracy of the drone diameter measurements
The conclusion
The results show that our drone provides accurate results for measuring more Belgian Mums in a significantly shorter period of time. However, the application is not yet complete for Kwekerij Baas.
Because some of the the Chrysantemums are sphere shaped, it is sometimes difficult to estimate the correct intersection where the diameter should be measured. In order to provide these results, we will develop the necessary algorithms with our software partners to fit their needs.
We are always open to do a validation check with customers to prove the accuracy of our drone. We’re constantly developing and improving our applications. Do you want us to validate an application as well?
The drone method
To validate the measurements, we took some representative chrysanthemums from the bay, as well as very not-representative ones amounting to a total of 20 plants. We placed QR codes on the ground as well as halfway the height of the chrysanthemum sphere. This last height is the height at which the diameter of the chrysanthemum is largest, and this is the height at which we measured the diameter of the sphere in this case.
When we take an image with the drone from above, we can calculate the actual size of each pixel in the photo in millimeters because we know the width and height of the QR codes beforehand. The amount of pixels of the diameter of the chrysanthemum multiplied by the amount of millimeters per pixel gives the actual size of the chrysanthemum diameter.
The test
To put the application to the test, we flew the drone over the selected chrysanthemums to collect the raw images. To validate the method, we also recorded manual diameter measurements, to get an idea of how well these measurements would correspond to the drone and QR code measurements. 5 different people measured each plant manually with measuring tape.
The results
In this table you can see the results of the test. We measured 20 plants in total. Of these 20 plants, number 2 and 11 (highlighted) were not spherically shaped, so the diameter of these chrysantemums is still under discussion.
Hand measurement
In the second column you can see the hand measurements for each plant. The value displayed for each plant is the average of the 5 manually measured values. It was interesting to see the amount of spread in measurements that the people would gather. It sometimes even ranged 5 cm. Also with repeated measurements of the same chrysanthemum, people would sometimes measure different values.
QR diameter
The third column is QR diameter. This is the length of the diameter of the smallest circle that you could possibly fit around the detected chrysantemum. This diameter is based on the pixel size of the QR code height and width that is placed next to the plant on the widest height. In cases where the plant is not spherically shaped our software makes a sphere of the smallest circle of the irregular plant.
Diameter inner circle
The fourth column is the drone measurement of the same diameter, but this time based on the height that the drone reports with each image. This is the measurement that we want to use for this application. Ideally, QR diameter and diameter_inner_circle_inner match, but this depends on many circumstances in our validation setup.
Code measurements
Code measurements is the length of the diameter that has been manually annotated on the drone images by a person. The QR code next to it has been manually annotated as well. This approach has been applied as well to compare the different methods: drone height based approach and deep learning detection of the largest diameter of the chrysantemum.
Differences QR annotation
The 6th and 7th column show the absolute difference between the drone height approach and QR diameter, and the absolute difference between the Manual annotation approach and the QR diameter. As mentioned, we expect the QR method to represent the measurements closest to ground-truth. Ideally, the error between the drone and QR measurements is zero.
Differences hand annotation
The 8th and 9th column show the same differences, but then regarding differences between Manual plant measurements and not QR diameter.
Average error
Underneath you can find the average error, the minimum error and the maximum error for each of these last 4 columns. The highest error having a red color, and the lowest errors having a green color. From this, you can see that the drone measurement compares best with the QR diameter, while the manual annotation of the diameter corresponds better to the average of the manual measurements.
The minimum errors are very acceptable (around 1mm), while the max errors of all comparisons are quite large (around 5 cm). If we would exclude measurement 2 and 11 from the data, the average and max errors drop significantly.
Final outcome
Our drone measured these 20 plants in 1 minute while it took 15-20 minutes to measure them by hand. Also when hand measurements are done only about 5 out of 50 plants are measured while our drone measures every single plant in the bay.
The problem that emerged from this test is that our drone does not measure each chrysantemum accurately. A difference of 2 cm from the measurement to the manually measured plant is a lot.
We investigated wheather this had to do with the sphereness of the Belgian Mums, but there was no strong correlation.
Our next step is to fly the drone over one or multiple bays each week, and record manual measurements correspondingly. This will provide us with a time series of drone and manual measurements that will show whether the diameter measurements of the drone are in line with human expectations.
Corvus Drones will now improve the measuring application to fit the wishes of Kwekerij Baas.