fibber.benchmark.benchmark_style_transfer module

class fibber.benchmark.benchmark_style_transfer.Benchmark(output_dir, dataset_name, trainset=None, testset=None, attack_set=None, subsample_attack_set=0, subsample_offset=0, customized_clf=None, enable_bert_clf=True, use_gpu_id=- 1, gpt2_gpu_id=- 1, bert_gpu_id=- 1, bert_ppl_gpu_id=- 1, ce_gpu_id=- 1, bert_clf_steps=20000, bert_clf_bs=32)[source]

Bases: object

Benchmark framework for adversarial attack methods on text classification.

Initialize Benchmark framework.

Parameters

  • output_dir (str) – the directory to write outputs including model, sentences, metrics and log.

  • dataset_name (str) – the name of the dataset.

  • trainset (dict) – the training set. If the dataset_name matches built-in datasets, trainset should be None.

  • testset (dict) – the test set. If the dataset_name matches built-in datasets, testset should be None.

  • attack_set (dict or None) – the set to run adversarial attack. Use None to attack the testset.

  • subsample_attack_set (int) – subsample the attack set. 0 to use the whole attack set.

  • customized_clf (MetricBase) – an classifier object.

  • enable_bert_clf (bool) – whether to enable bert classifier in metrics. You can disable it when you are attacking your own classifier.

  • use_gpu_id (int) – the gpu to run universal sentence encoder to compute metrics. -1 for CPU.

  • gpt2_gpu_id (int) – the gpu to run the GPT2-medium language model to compute metrics. -1 for CPU.

  • bert_gpu_id (int) – the gpu to run the BERT text classifier, which is the model being attacked. -1 for CPU.

  • bert_clf_steps (int) – number of steps to train the BERT text classifier.

  • bert_clf_bs (int) – the batch size to train the BERT classifier.

get_metric_bundle()[source]
run_benchmark(paraphrase_strategy='IdentityStrategy', strategy_gpu_id=- 1, num_paraphrases_per_text=50, exp_name=None, update_global_results=False)[source]

Run the benchmark.

Parameters

  • paraphrase_strategy (str or StrategyBase) – the paraphrase strategy to benchmark. Either the name of a builtin strategy or a customized strategy derived from StrategyBase.

  • strategy_gpu_id (int) – the gpu id to run the strategy. -1 for CPU. Ignored when paraphrase_strategy is an object.

  • num_paraphrases_per_text (int) – number of paraphrases for each sentence.

  • exp_name (str or None) – the name of current experiment. None for default name. the default name is <dataset_name>-<strategy_name>-<date>-<time>.

  • update_global_results (bool) – whether to write results in <fibber_root_dir> or the benchmark output dir.

Returns

A dict of evaluation results.

fibber.benchmark.benchmark_style_transfer.get_strategy(arg_dict, dataset_name, strategy_name, strategy_gpu_id, output_dir, metric_bundle)[source]

Take the strategy name and construct a strategy object.

fibber.benchmark.benchmark_style_transfer.main()[source]

previous

fibber.benchmark.benchmark_adversarial_attack module

next

fibber.benchmark.benchmark_utils module