Detection results

Author

Janne Mäyrä

Published

October 4, 2024

Code 
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import geopandas as gpd
from pathlib import Path
import os
import numpy as np
import fiona 
import json
import warnings
warnings.filterwarnings("ignore")

import sys
sys.path.insert(1, '..')
from src.colors import *

model_path = Path('../runs_2023')

Cross-validation

Evaluate the cross-validation performance of different YOLOv8 models.

Metrics used here are the following:

  • \(Precision = \frac{tp}{tp+fp}\), with IoU-threshold of 0.5
  • \(Recall = \frac{tp}{tp+fn}\), with IoU-threshold of 0.5
  • \(mAP50\) is the area under the precision-recall curve with IoU threshold of 0.5
  • \(mAP\) or \(AP@[.5:.95]\) is the average \(mAP\) for IoU from 0.5 to 0.95 with a step size of 0.05
  • \(IoU\) corresponds to Intersection-over-Union, the ratio between the overlap and union of two bounding boxes

All models were trained for maximum 200 epochs, with patience=30 monitoring mAP.

Figure 1: 5-fold crossvalidation results for different model architectures.
Code 
resdf.pivot_table(values=['Precision', 'Recall', 'mAP50', 'mAP'], index=['Model'], 
                  aggfunc={'Precision':[min, max, np.mean],
                           'Recall':[min, max, np.mean],
                           'mAP50':[min, max, np.mean],
                           'mAP':[min, max, np.mean]})
Precision Recall mAP mAP50
max mean min max mean min max mean min max mean min
Model
yolov8l 0.85450 0.846998 0.83973 0.84975 0.830720 0.82398 0.41694 0.411712 0.40505 0.87459 0.863334 0.85743
yolov8m 0.83592 0.827938 0.81134 0.84939 0.832380 0.80153 0.41739 0.410742 0.40496 0.87071 0.860378 0.84545
yolov8n 0.84688 0.823160 0.81273 0.79587 0.783928 0.76895 0.38816 0.380828 0.37637 0.84323 0.827192 0.81776
yolov8s 0.86363 0.834556 0.80835 0.83639 0.825554 0.81469 0.41098 0.406258 0.40374 0.86933 0.858632 0.84776
yolov8x 0.84028 0.834728 0.83019 0.84891 0.828462 0.80858 0.42065 0.411532 0.40231 0.86203 0.858136 0.85131

Visual comparisation of results

Figure 2: Example result comparison for different models

Test set results

The predictions for the test set are cleaned using the following steps:

  1. All prediction whose centroid points are not located on water are discarded. The water mask used contains layers jarvi (Lakes), meri (Sea) and virtavesialue (Rivers as polygon geometry) from the Topographic database by the National Land Survey of Finland. Unfortunately this also discards all points not within the Finnish borders. (The National Land Survey of Finland Topographic Database 04/2023, license CC 4.0 BY)
  2. All predictions whose centroid points are located on water rock areas are discarded. The mask is the layer vesikivikko (Water rock areas) from the Topographic database.
  3. All predictions that contain an above water rock within the bounding box are discarded. The mask contains classes 38511, 38512, 38513 from the layer vesikivi in the Topographical database.
  4. All predictions that contain a lighthouse or a sector light within the bounding box are discarded. Lighthouses and sector lights come from Finnish Transport Infrastructure Agency, ty_njr class ids are 1, 2, 3, 4, 5, 8. (Finnish Transport Agency / Open API, license CC 4.0 BY)
  5. All predictions that are wind turbines, found in Topographic database layer tuulivoimalat
  6. TODO Filter aquaculture and net pens as soon as suitable layer for them is found
  7. All predictions that are obviously too large are discarded. The prediction is defined to be “too large” if either of its edges is longer than 750 meters.

Topographic database can be acquired here, and the data from Finnish Transport Infrastructure Agency is available here (Maritime transport, Aids to navigation).

Evaluate the results with COCO metrics. True positives are defined based on IoU of predicted polygons using shapely.

Figure 3: Test set results by model architecture.
Code 
resdf.pivot_table(values=['Precision', 'Recall', 'mAP50', 'mAP'], index=['Model'], 
                  aggfunc={'Precision':[min, max, np.mean],
                           'Recall':[min, max, np.mean],
                           'mAP50':[min, max, np.mean],
                           'mAP':[min, max, np.mean]})
Precision Recall mAP mAP50
max mean min max mean min max mean min max mean min
Model
yolov8l 0.877358 0.863431 0.853155 0.863636 0.856511 0.851966 0.463503 0.455233 0.446569 0.896509 0.882343 0.873198
yolov8m 0.888228 0.867010 0.846107 0.875307 0.852703 0.824939 0.464857 0.455235 0.443199 0.903983 0.888179 0.871075
yolov8n 0.840748 0.827731 0.815111 0.838452 0.821499 0.800983 0.417369 0.407895 0.398058 0.856725 0.847745 0.836608
yolov8s 0.869427 0.858166 0.848823 0.860565 0.847666 0.838452 0.448884 0.435405 0.427492 0.886253 0.878227 0.867800
yolov8x 0.867101 0.858786 0.841434 0.859951 0.846929 0.817568 0.455922 0.452866 0.446565 0.886113 0.879400 0.865797

Best performing model for each architecture is available on https://huggingface.co/mayrajeo/marine-vessel-detection.

Code 
resdf
Model fold Precision Recall mAP50 mAP
0 yolov8n <_io.TextIOWrapper name='../runs_2023/yolov8n_... 0.835373 0.838452 0.856328 0.410469
1 yolov8n <_io.TextIOWrapper name='../runs_2023/yolov8n_... 0.820560 0.828624 0.837642 0.398058
2 yolov8n <_io.TextIOWrapper name='../runs_2023/yolov8n_... 0.815111 0.815111 0.836608 0.403318
3 yolov8n <_io.TextIOWrapper name='../runs_2023/yolov8n_... 0.840748 0.800983 0.856725 0.410264
4 yolov8n <_io.TextIOWrapper name='../runs_2023/yolov8n_... 0.826864 0.824324 0.851424 0.417369
5 yolov8s <_io.TextIOWrapper name='../runs_2023/yolov8s_... 0.855296 0.853194 0.881689 0.434618
6 yolov8s <_io.TextIOWrapper name='../runs_2023/yolov8s_... 0.869427 0.838452 0.886253 0.448884
7 yolov8s <_io.TextIOWrapper name='../runs_2023/yolov8s_... 0.858302 0.844595 0.877176 0.437534
8 yolov8s <_io.TextIOWrapper name='../runs_2023/yolov8s_... 0.858982 0.860565 0.878220 0.428499
9 yolov8s <_io.TextIOWrapper name='../runs_2023/yolov8s_... 0.848823 0.841523 0.867800 0.427492
10 yolov8m <_io.TextIOWrapper name='../runs_2023/yolov8m_... 0.855055 0.862408 0.881303 0.443199
11 yolov8m <_io.TextIOWrapper name='../runs_2023/yolov8m_... 0.873036 0.853194 0.891204 0.461821
12 yolov8m <_io.TextIOWrapper name='../runs_2023/yolov8m_... 0.846107 0.847666 0.871075 0.446552
13 yolov8m <_io.TextIOWrapper name='../runs_2023/yolov8m_... 0.872627 0.875307 0.893329 0.459746
14 yolov8m <_io.TextIOWrapper name='../runs_2023/yolov8m_... 0.888228 0.824939 0.903983 0.464857
15 yolov8l <_io.TextIOWrapper name='../runs_2023/yolov8l_... 0.853155 0.863636 0.874179 0.449165
16 yolov8l <_io.TextIOWrapper name='../runs_2023/yolov8l_... 0.859519 0.856880 0.877627 0.446569
17 yolov8l <_io.TextIOWrapper name='../runs_2023/yolov8l_... 0.854244 0.853194 0.873198 0.454970
18 yolov8l <_io.TextIOWrapper name='../runs_2023/yolov8l_... 0.877358 0.856880 0.896509 0.461961
19 yolov8l <_io.TextIOWrapper name='../runs_2023/yolov8l_... 0.872876 0.851966 0.890204 0.463503
20 yolov8x <_io.TextIOWrapper name='../runs_2023/yolov8x_... 0.865801 0.859951 0.886113 0.455020
21 yolov8x <_io.TextIOWrapper name='../runs_2023/yolov8x_... 0.863158 0.856265 0.886026 0.453919
22 yolov8x <_io.TextIOWrapper name='../runs_2023/yolov8x_... 0.841434 0.850737 0.865797 0.446565
23 yolov8x <_io.TextIOWrapper name='../runs_2023/yolov8x_... 0.856436 0.850123 0.874200 0.455922
24 yolov8x <_io.TextIOWrapper name='../runs_2023/yolov8x_... 0.867101 0.817568 0.884866 0.452905