Model testingΒΆ
Pipeline testing the performance of different models on the NW sample.
import numpy
import tensorflow as tf
from model_builder import model_builder, relabel, class_merger, balancer
import model_builder
import tools_keras
from itertools import product
Load chips
specs = {
'tensor': (
'../urbangrammar_samba/'
'spatial_signatures/chips/sample.npz'
),
'folder': "../urbangrammar_samba/spatial_signatures/ai/nw_32/",
}
data = numpy.load(specs["tensor"], allow_pickle=True)
chips = data["chips"]
labels = data["labels"]
Shuffle data to ensure full randomness before splitting to test, validation and secret.
numpy.random.seed(0)
shuffled_idx = numpy.arange(0, chips.shape[0])
numpy.random.shuffle(shuffled_idx)
chips = chips[shuffled_idx]
labels = labels[shuffled_idx]
numpy.unique(labels, return_counts=True)
(array(['0_0', '1_0', '2_0', '2_1', '2_2', '3_0', '4_0', '5_0', '6_0',
'7_0', '8_0', '9_0', '9_2', '9_4'], dtype=object),
array([25571, 1576, 379, 243, 255, 4827, 3090, 2114, 271,
26610, 76, 18, 11, 1]))
Merge groups
group_mapping = [
['9_0', '9_1', '9_2', '9_4', '9_5', '2_0', '2_1', '2_2'],
['1_0', '3_0', '5_0', '6_0', '8_0'],
['0_0', '4_0', '7_0']
]
labels = class_merger(labels, group_mapping)
# If you don't want to merge groups, use `relabel` to ensure your labels are in the proper format
# labels = relabel(labels)
Drop chips from excessive classes
mask = balancer(labels, max_ratio=10)
Total number of selected chips: 18841 out of 65042
Counts:
{0: 907, 1: 8864, 2: 9070}
labels = labels[mask]
chips = chips[mask]
Rescale
chips = chips / (chips.max() / 255.0)
Split data into train and validation.
batch_size = 32
split_1 = int(chips.shape[0] * 0.6)
split_2 = int(chips.shape[0] * 0.8)
# assert that all labels are present in train and validation datasets
assert (numpy.unique(labels[:split_1]) == numpy.unique(labels[split_1:split_2])).all()
assert (numpy.unique(labels[:split_1]) == numpy.unique(labels[split_2:])).all()
train_dataset = tf.data.Dataset.from_tensor_slices((chips[:split_1], labels[:split_1]))
validation_dataset = tf.data.Dataset.from_tensor_slices((chips[split_1:split_2], labels[split_1:split_2]))
secret_dataset = tf.data.Dataset.from_tensor_slices((chips[split_2:], labels[split_2:]))
train_dataset = train_dataset.batch(batch_size=batch_size)
validation_dataset = validation_dataset.batch(batch_size=batch_size)
secret_dataset = secret_dataset.batch(batch_size=batch_size)
Create model and fit in a loop over options.
model_specs = {
'meta_class_map': group_mapping,
'meta_class_names': ["centres", "periphery", "countryside"],
'meta_chip_size': 8,
}
for model_name, bridge, neurons in product(["vgg19", "efficientnet"], ["pooling"], [256]):
model = model_builder.model_builder(
model_name=model_name,
bridge=bridge,
top_layer_neurons=neurons,
n_labels=3,
input_shape=(8, 8, 3),
)
h = tools_keras.fit_phase(
model,
train_dataset,
validation_dataset,
secret_dataset,
log_folder=specs["folder"] + "logs",
pred_folder=specs["folder"] + "pred",
model_folder=specs["folder"] + "model",
json_folder=specs["folder"] + "json",
specs=model_specs,
epochs=250,
patience=5,
verbose=True,
)
print(model_name, bridge, neurons, "done")
Model: "vgg19_pooling_256_3"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_14 (InputLayer) [(None, 8, 8, 3)] 0
_________________________________________________________________
resizing_6 (Resizing) (None, 224, 224, 3) 0
_________________________________________________________________
tf.__operators__.getitem_2 ( (None, 224, 224, 3) 0
_________________________________________________________________
tf.nn.bias_add_2 (TFOpLambda (None, 224, 224, 3) 0
_________________________________________________________________
vgg19 (Functional) (None, 7, 7, 512) 20024384
_________________________________________________________________
global_average_pooling2d_6 ( (None, 512) 0
_________________________________________________________________
dense_12 (Dense) (None, 256) 131328
_________________________________________________________________
dense_13 (Dense) (None, 3) 771
=================================================================
Total params: 20,156,483
Trainable params: 132,099
Non-trainable params: 20,024,384
_________________________________________________________________
None
2021-12-17 16:12:36.273098: I tensorflow/core/profiler/lib/profiler_session.cc:131] Profiler session initializing. 2021-12-17 16:12:36.273153: I tensorflow/core/profiler/lib/profiler_session.cc:146] Profiler session started. 2021-12-17 16:12:36.482919: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session tear down. 2021-12-17 16:12:36.483057: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1749] CUPTI activity buffer flushed
Epoch 1/250
1/2565 [..............................] - ETA: 32:20 - loss: 1.7970 - accuracy: 0.2500
2021-12-17 16:12:37.487387: I tensorflow/core/profiler/lib/profiler_session.cc:131] Profiler session initializing. 2021-12-17 16:12:37.487419: I tensorflow/core/profiler/lib/profiler_session.cc:146] Profiler session started.
2/2565 [..............................] - ETA: 17:08 - loss: 1.4709 - accuracy: 0.3438
2021-12-17 16:12:38.528309: I tensorflow/core/profiler/lib/profiler_session.cc:66] Profiler session collecting data. 2021-12-17 16:12:38.533468: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1749] CUPTI activity buffer flushed 2021-12-17 16:12:38.623424: I tensorflow/core/profiler/internal/gpu/cupti_collector.cc:673] GpuTracer has collected 176 callback api events and 173 activity events. 2021-12-17 16:12:38.635550: I tensorflow/core/profiler/lib/profiler_session.cc:164] Profiler session tear down. 2021-12-17 16:12:38.696224: I tensorflow/core/profiler/rpc/client/save_profile.cc:136] Creating directory: ../urbangrammar_samba/spatial_signatures/ai/nw_8/logs/vgg19_pooling_256_3/train/plugins/profile/2021_12_17_16_12_38 2021-12-17 16:12:38.715968: I tensorflow/core/profiler/rpc/client/save_profile.cc:142] Dumped gzipped tool data for trace.json.gz to ../urbangrammar_samba/spatial_signatures/ai/nw_8/logs/vgg19_pooling_256_3/train/plugins/profile/2021_12_17_16_12_38/985b7fd52aa6.trace.json.gz
3/2565 [..............................] - ETA: 31:25 - loss: 1.3250 - accuracy: 0.3854
2021-12-17 16:12:38.738771: I tensorflow/core/profiler/rpc/client/save_profile.cc:136] Creating directory: ../urbangrammar_samba/spatial_signatures/ai/nw_8/logs/vgg19_pooling_256_3/train/plugins/profile/2021_12_17_16_12_38 2021-12-17 16:12:38.753362: I tensorflow/core/profiler/rpc/client/save_profile.cc:142] Dumped gzipped tool data for memory_profile.json.gz to ../urbangrammar_samba/spatial_signatures/ai/nw_8/logs/vgg19_pooling_256_3/train/plugins/profile/2021_12_17_16_12_38/985b7fd52aa6.memory_profile.json.gz 2021-12-17 16:12:38.796317: I tensorflow/core/profiler/rpc/client/capture_profile.cc:251] Creating directory: ../urbangrammar_samba/spatial_signatures/ai/nw_8/logs/vgg19_pooling_256_3/train/plugins/profile/2021_12_17_16_12_38 Dumped tool data for xplane.pb to ../urbangrammar_samba/spatial_signatures/ai/nw_8/logs/vgg19_pooling_256_3/train/plugins/profile/2021_12_17_16_12_38/985b7fd52aa6.xplane.pb Dumped tool data for overview_page.pb to ../urbangrammar_samba/spatial_signatures/ai/nw_8/logs/vgg19_pooling_256_3/train/plugins/profile/2021_12_17_16_12_38/985b7fd52aa6.overview_page.pb Dumped tool data for input_pipeline.pb to ../urbangrammar_samba/spatial_signatures/ai/nw_8/logs/vgg19_pooling_256_3/train/plugins/profile/2021_12_17_16_12_38/985b7fd52aa6.input_pipeline.pb Dumped tool data for tensorflow_stats.pb to ../urbangrammar_samba/spatial_signatures/ai/nw_8/logs/vgg19_pooling_256_3/train/plugins/profile/2021_12_17_16_12_38/985b7fd52aa6.tensorflow_stats.pb Dumped tool data for kernel_stats.pb to ../urbangrammar_samba/spatial_signatures/ai/nw_8/logs/vgg19_pooling_256_3/train/plugins/profile/2021_12_17_16_12_38/985b7fd52aa6.kernel_stats.pb
2565/2565 [==============================] - 333s 129ms/step - loss: 0.8870 - accuracy: 0.5651 - val_loss: 0.8614 - val_accuracy: 0.5747
Epoch 2/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8590 - accuracy: 0.5799 - val_loss: 0.8534 - val_accuracy: 0.5758
Epoch 3/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8506 - accuracy: 0.5838 - val_loss: 0.8486 - val_accuracy: 0.5765
Epoch 4/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8461 - accuracy: 0.5851 - val_loss: 0.8454 - val_accuracy: 0.5801
Epoch 5/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8432 - accuracy: 0.5866 - val_loss: 0.8424 - val_accuracy: 0.5848
Epoch 6/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8400 - accuracy: 0.5878 - val_loss: 0.8415 - val_accuracy: 0.5837
Epoch 7/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8382 - accuracy: 0.5891 - val_loss: 0.8422 - val_accuracy: 0.5842
Epoch 8/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8363 - accuracy: 0.5896 - val_loss: 0.8401 - val_accuracy: 0.5836
Epoch 9/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8341 - accuracy: 0.5910 - val_loss: 0.8395 - val_accuracy: 0.5849
Epoch 10/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8325 - accuracy: 0.5916 - val_loss: 0.8390 - val_accuracy: 0.5842
Epoch 11/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8309 - accuracy: 0.5918 - val_loss: 0.8398 - val_accuracy: 0.5821
Epoch 12/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8295 - accuracy: 0.5934 - val_loss: 0.8389 - val_accuracy: 0.5841
Epoch 13/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8281 - accuracy: 0.5931 - val_loss: 0.8386 - val_accuracy: 0.5861
Epoch 14/250
2565/2565 [==============================] - 334s 130ms/step - loss: 0.8265 - accuracy: 0.5943 - val_loss: 0.8384 - val_accuracy: 0.5864
Epoch 00014: early stopping
time elapsed: 4698.6s
prediction saved
INFO:tensorflow:Assets written to: ../urbangrammar_samba/spatial_signatures/ai/nw_8/model/vgg19_pooling_256_3/assets
perf_model_accuracy for train: 0.5947611927986145
# tools_keras.flush(specs["folder"])
chips.shape
(102234, 15, 15, 3)