Benchmark experiments for the naive Bayes classifier show that the inclusion of all features does not yield a statistically significant improvement in classification accuracy.

• Run 1 - 33 / 40 Correctly Classified Sweeps => 82.5% [output][result]
• Run 2 - 34 / 40 Correctly Classified Sweeps => 85.0% [output][result]
• Run 3 - 34 / 40 Correctly Classified Sweeps => 85.0% [output][result]

Average Accuracy: 84.17%

 Naive Bayes benchmarks on crane show no major improvement. The experiments were run without subsampling the data and the best known subset of features for naive Bayes was used (Range, Reflectivity, Velocity, Spectrum Width, Reflectivity Kurtosis, Reflectivity Skewness, and Velocity Skewness).

• Run 1 - 33 / 40 Correctly Classified Sweeps => 82.5% [output][report]
• Run 2 - 33 / 40 Correctly Classified Sweeps => 82.5% [output][report]
• Run 3 - 34 / 40 Correctly Classified Sweeps => 85.0% [output][report]

Average Accuracy: 83.33%

 The results from the NN benchmark experiments on Crane are in. Three tests were run, each on 40 training files divided into 10 folds for 10 fold cross validation. No sub sampling was done on the data and all 14 features were included. The neural nets were trained over the course of 3000 epochs, with the following results:

• Run 1 - 35 / 40 Correctly Classified Sweeps => 87.5%  [output] [report]
• Run 2 - 37 / 40 Correctly Classified Sweeps => 92.5%  [output] [report]
• Run 3 - 37 / 40 Correctly Classified Sweeps => 92.5%  [output] [report]

       Average Accuracy : 90.83%

These results are better than the original neural network benchmark test which was lost due to corruption.
       

I am currently running a benchmark experiment in which I am training a neural network on all 14 features without doing any sub-sampling of the data. This is a repeat of an experiment that was previously run, but was corrupted.

This experiment should take several hours to finish, but the current progress can be monitored here.

Manuel Suarez was kind enough to provide us with some more training data. His data mostly comes from the Milwaukee area. We specifically asked for data that would be harder to classify than the cut and dry sweeps in our original training data set.

The results of the first experiment confirm that this data will be harder to classify. The Suarez dataset consists of 22 sweeps and when trained on the original Diehl dataset, the neural network classified the Suarez dataset with an accuracy of only 35%. As this is worse than guessing, it would seem there are elements in this dataset that the classifier must be exposed to during training.

Training and validating using only this dataset produced slightly better results, coming in at an accuracy of 64%. Training and testing using both datasets absorbed some of this variability and produced a mean classification accuracy of 84.5% with a 95% confidence interval of (79%, 90%).