fibber.metrics.attack_aggregation_utils module¶
This module defines customized metric aggregation functions.
-
fibber.metrics.attack_aggregation_utils.add_sentence_level_adversarial_attack_metrics(metric_bundle, best_adv_metric_name=None, best_adv_metric_lower_better=None, clf_agg_type='worst')[source]¶ Add advanced aggregation functions related to adversarial attack to a specific metric bundle.
- Parameters
metric_bundle (MetricBundle) – a metric bundle object to add aggregation functions.
best_adv_metric_name (str) –
best_adv_metric_lower_better (bool) –
-
fibber.metrics.attack_aggregation_utils.get_best_adv_by_metric(data_record, target_clf, metric_name, lower_better)[source]¶ Find the best adversarial example by some metric.
- Parameters
data_record (dict) – a data record with paraphrases and metrics.
target_clf (str) – the targeted classifier.
metric_name (str) – the metric to pick the best adversarial example.
lower_better (bool) – if true, find the adversarial example with the smallest metric value.
- Returns
- the metrics of the best adversarial example. None if no legitimate adversarial
example is found or the original sentence is misclassified.
- Return type
(dict)
-
fibber.metrics.attack_aggregation_utils.get_best_adv_metric_fn_constructor(get_best_adv_fn, metric_name, target_clf)[source]¶ Returns an aggregation function that extracts the value of a specified metric for the best adversarial example.
The aggregation function returns NaN if no legitimate adversarial example is found.
- Parameters
get_best_adv_fn (fn) – a function that returns the metric dict of the best adversarial example.
metric_name (str) – a metric name.
target_clf (str) – the targeted classifier.
- Returns
an aggregation function that takes data_record as an input.
- Return type
(fn)
-
fibber.metrics.attack_aggregation_utils.paraphrase_classification_accuracy_agg_fn_constructor(target_clf, type)[source]¶ This function makes a aggregation function for the target classification metric.
The aggregation function outputs the after attack accuracy of the BERT classifier.
- Parameters
target_clf (str) – the metric name of the target classifier.