About Fingerprint Testing¶

A fingerprint is a hash value, which is calculated during a simulation run from specified “ingredients”, i.e. properties of the simulation such as time of events, module names, packet data, etc. The hash is updated at each event which takes part in the fingerprint calculation (as defined by the ingredients, filtering, etc) until the end of the simulation (or some defined time limit), resulting in a fingerprint value. This value is characteristic of the simulation’s trajectory; a fingerprint change indicates a change in the trajectory.

The fingerprint contains a 32-bit hexadecimal hash value and the ingredients used for calculating it, denoted by letters. Here is an example:

53de-64a7/tplx

In this example, t means time of event, p the full path of modules, l the message length, and x the extra data added programatically. Other possible ingredients include various properties of messages, modules and results. For the complete list, check out the Fingerprint section in the OMNeT++ Simulation Manual.

The fingerprints can be specified in the ini file for each configuration. Specifying any fingerprint enables fingerprint calculation for the simulation. For example:

[Config Ethernet]
network = Wired
fingerprint = 0000-0000/tplx

The fingerprint will be calculated for that simulation when run in Qtenv or Cmdenv. Qtenv and Cmdenv indicate whether the fingerprint is verified, and print the calculated fingerprint value. The original value can be replaced with the resulting one in the ini file.

INET’s fingerprint test tool automates this process, making it easier to work with many simulations and fingerprints.

The fingerprint tool¶

The fingerprint tool is a conventient way to run fingerprint tests. It is located in the inet/bin folder, and when the inet directory is added to the PATH, it can be run from any project directory.

The fingerprint test tool uses .csv files to run fingerprint tests./uses .csv files to keep track of test runs A line in the .csv file defines a simulation run by specifying the working directory, command line arguments, sim time limit, fingerprint+ingredients, expected result, and tags. The result of one fingerprint test can either be PASS, FAILED or ERROR. A test results in FAILED if the calculated fingerprint differs from the one in the .csv file; it results in ERROR when there was an error during the simulation.

When run without arguments, the fingerprint test tool runs all tests in all .csv files in the current directory. Specific .csv files to run can be selected with the first argument. Also, the set of tests to run can be filtered with the -m command line option, by providing a regex expression/regex that is matched against all information in the .csv files.

The fingerprint test tool runs tests concurrently by default, using all CPU cores.

For the list of all available options, run the tool with the -h argument.

When the tests are finished, the tool may create additional .csv files (appending UPDATED, FAILED, and ERROR to the .csv file’s name):

an UPDATED file, which has the fingerprints just calculated for all lines
a FAILED file, which contains just the lines for the failed simulations, with the calculated fingerprint. This file is useful for re-running just the simulations with failed fingerprints
an ERROR file, which contains the lines for simulations which finished with errors, making it easier to rerun just those simulations

The updated file can be used to overwrite the original one to accept the new fingerprints.

The configurations¶

The tutorial contains several simulations, which will be used to demonstrate how various changes affect fingerprints. Here is an overview of the simulations:

Ethernet: Contains an Ethernet network with wired hosts and routers. The server host sends UDP packets to two of the hosts:
Wifi: Contains two ad-hoc hosts; one of them sends UDP packets to the other:
EthernetShortPacket: The same as the Ethernet configuration, but the server sends 10-Byte UDP packets.
WifiShortPacket: The same as the Wifi configuration, but the host sends 10-Byte UDP packets.

The baseline.csv file in the tutorial’s directory containing the fingerprints:

# working directory, command line arguments, simulation time limit, fingerprint, expected result, tags

.,        -f omnetpp.ini -c Ethernet -r 0,             0.2s,         4500-0673/tplx, PASS, EasyToHandleChanges RenamingSubmodule RenamingParameter ChangingPacketLength AddingNewEvents1 AddingNewEvents2 AcceptingFPChanges
.,        -f omnetpp.ini -c EthernetShortPacket -r 0,  0.2s,         ea97-154f/tplx, PASS, ChangingPacketLength
.,        -f omnetpp.ini -c Wifi -r 0,                   5s,         791d-aba6/tplx, PASS, EasyToHandleChanges RenamingSubmodule ChangingPacketLength ChangingTimer AddingNewEvents1 AddingNewEvents2 AcceptingFPChanges
.,        -f omnetpp.ini -c WifiShortPacket -r 0,        5s,         d801-fc01/tplx, PASS, ChangingPacketLength

The .csv file contains the correct fingerprints for the latest INET version, i.e. they all pass (the actual values in the baseline.csv might differ from the values in the tutorial text). To try the examples, the user is expected to make the changes described in the tutorial (or experiment with others) and run the fingerprint tests to see how the model changes affect the fingerprints.

Note

Before trying the example in a step, reset the model changes of the previous step.

Running the tests¶

The fingerprints can be run from the command line. Make sure to run . setenv in both the omnetpp and inet folders:

$ cd ~/workspace/omnetpp
$ . setenv
$ cd ~/workspace/inet
$ . setenv
$ cd ~/workspace/inet/tutorials/fingerprint
$ inet_fingerprinttest

The inet_fingerprinttest runs all simulations specified in all .csv files in the currect folder:

$ inet_fingerprinttest
. -f omnetpp.ini -c Ethernet -r 0 ... : PASS
. -f omnetpp.ini -c EthernetShortPacket -r 0 ... : PASS
. -f omnetpp.ini -c Wifi -r 0 ... : PASS
. -f omnetpp.ini -c WifiShortPacket -r 0 ... : PASS

----------------------------------------------------------------------
Ran 4 tests in 12.451s

OK

Log has been saved to test.out
Test results equals to expected results

Each simulation in baseline.csv has tags specified indicating which step it is relevant for, so that only the relevant simulations can be run for each step. The tags can be filtered for with the -m command line arguments. For example, running the simulations for the Changing a Timer step:

$ inet_fingerprinttest -m ChangingTimer
. -f omnetpp.ini -c Wifi -r 0 ... : PASS

----------------------------------------------------------------------
Ran 1 test in 8.013s

OK

Log has been saved to test.out
Test results equals to expected results

The tags to use is indicated at each step.

Baseline ingredients¶

The baseline fingerprint ingredients are chosen because they are sensitive to a broad range of changes in the model (the default ingredients in fingerprint tests in INET is often tplx). If the ingredients are too sensitive, even trivial changes might lead to failed tests; if the ingredients are not sensitive enough, they might fail to detect regressions.

The workflow¶

The tutorial describes the workflow of making typical changes and verifying the model. Generally, the workflow involves a change in the model, which we suspect will not invalidate it. We also know that the baseline fingerprints are sensitive to this change; we choose other ingredients and create alternative fingerprints which are not sensitive to the particular change, but sensitive to others. Then, after making the model change, the fingerprint tests should pass, and the model should be verified. When the model is verified, the baseline ingredients can be restored.

Note

Fingerprint testing depends on repeatable deterministic simulations, random number generation, etc. If the tests are not repeatable, i.e. they don’t result in the same fingerprints when run multiple times (e.g. emulation), fingerprint tests are useless.