About Fingerprint Testing

A fingerprint is a hash value, which is calculated during a simulation run from specified “ingredients”, i.e. properties of the simulation such as time of events, module names, packet data, etc. The hash is updated at each event which takes part in the fingerprint calculation (as defined by the ingredients, filtering, etc) until the end of the simulation (or some defined time limit), resulting in a fingerprint value. This value is characteristic of the simulation’s trajectory; a fingerprint change indicates a change in the trajectory.

The fingerprint contains a 32-bit hexadecimal hash value and the ingredients used for calculating it, denoted by letters. Here is an example:

53de-64a7/tplx

In this example, t means time of event, p the full path of modules, l the message length, and x the extra data added programatically. Other possible ingredients include various properties of messages, modules and results. For the complete list, check out the Fingerprint section in the OMNeT++ Simulation Manual.

The fingerprints can be specified in the ini file for each configuration. Specifying any fingerprint enables fingerprint calculation for the simulation. For example:

[Config Ethernet]
network = Wired
fingerprint = 0000-0000/tplx

The fingerprint will be calculated for that simulation when run in Qtenv or Cmdenv. Qtenv and Cmdenv indicate whether the fingerprint is verified, and print the calculated fingerprint value. The original value can be replaced with the resulting one in the ini file.

INET’s fingerprint test commands automate this process, making it easier to work with many simulations and fingerprints.

The fingerprint tests

INET keeps the correct fingerprints in the JSON file tests/fingerprint/store.json. This file is the fingerprint store. A store entry defines one simulation run. It has the working directory, the configuration, the run number, the sim time limit, and the fingerprint with its ingredients. The store file is part of the INET repository. The tests of the tutorial use the entries of the tutorials/fingerprint working directory.

Two commands of opp_repl work with the store:

  • opp_run_fingerprint_tests runs the simulations. It compares the calculated fingerprints with the stored ones.

  • opp_update_fingerprint_test_results runs the simulations. It records the calculated fingerprints in the store.

Each test has one of these results: PASS, FAIL or ERROR. A test results in FAIL if the calculated fingerprint differs from the stored one. A test results in ERROR if there is an error during the simulation.

The commands run the tests at the same time. They use all CPU cores. To select tests, use the -w option for the working directory, the -c option for the configuration, and the -f option for a general filter. Each option takes a regular expression. The --ingredients option selects the fingerprint ingredients. The -m option selects the build mode, debug or release. The default is debug. Use the --help option to see all options.

The update command prints one line for each simulation:

  • KEEP means that the stored fingerprint is correct.

  • UPDATE means that the fingerprint is different. The command replaces the stored value.

  • INSERT means that the store has no value for these ingredients. The command adds the new value.

The update command changes the file tests/fingerprint/store.json. Use git diff tests/fingerprint/store.json to see the changes. To accept new fingerprints, run the update command and keep the changes of the store file. To reject them, restore the file with git checkout tests/fingerprint/store.json.

The configurations

The tutorial contains several simulations, which will be used to demonstrate how various changes affect fingerprints. Here is an overview of the simulations:

  • Ethernet: Contains an Ethernet network with wired hosts and routers. The server host sends UDP packets to two of the hosts:

    ../../../_images/EthernetNetwork.png
  • Wifi: Contains two ad-hoc hosts; one of them sends UDP packets to the other:

    ../../../_images/WifiNetwork.png
  • EthernetShortPacket: The same as the Ethernet configuration, but the server sends 10-Byte UDP packets.

  • WifiShortPacket: The same as the Wifi configuration, but the host sends 10-Byte UDP packets.

The fingerprint store tests/fingerprint/store.json contains the fingerprints of these four configurations. All four configurations have run number 0. The Ethernet and EthernetShortPacket configurations run for 0.2 s. The Wifi and WifiShortPacket configurations run for 5 s. The stored fingerprints use the tplx ingredients.

The store contains the correct fingerprints for the latest INET version, i.e. they all pass. To try the examples, make the changes that the tutorial describes, or try other changes. Then run the fingerprint tests to see how the changes affect the fingerprints.

Note

Before trying the example in a step, reset the model changes of the previous step.

Running the tests

Run the fingerprint tests from the command line. Make sure to run . setenv in both the omnetpp and inet folders:

$ cd ~/workspace/omnetpp
$ . setenv
$ cd ~/workspace/inet
$ . setenv
$ cd ~/workspace/inet/tutorials/fingerprint

The opp_run_fingerprint_tests command runs the simulations of the tutorial. Always give the project and the working directory filter. Use tplx as the ingredients:

$ opp_run_fingerprint_tests -w tutorials/fingerprint --ingredients tplx
tutorials/fingerprint -c Ethernet for 0.2s PASS
tutorials/fingerprint -c EthernetShortPacket for 0.2s PASS
tutorials/fingerprint -c Wifi for 5s PASS
tutorials/fingerprint -c WifiShortPacket for 5s PASS

Each step of the tutorial needs only some of the configurations. Use the -c option to run only those. For example, the Changing a Timer step needs the Wifi configuration only:

$ opp_run_fingerprint_tests -w tutorials/fingerprint --ingredients tplx -c '^Wifi$'
tutorials/fingerprint -c Wifi for 5s PASS

Each step shows the -c option to use.

Baseline ingredients

The baseline fingerprint ingredients are chosen because they are sensitive to a broad range of changes in the model (the default ingredients in fingerprint tests in INET is often tplx). If the ingredients are too sensitive, even trivial changes might lead to failed tests; if the ingredients are not sensitive enough, they might fail to detect regressions.

The workflow

The tutorial describes the workflow of making typical changes and verifying the model. Generally, the workflow involves a change in the model, which we suspect will not invalidate it. We also know that the baseline fingerprints are sensitive to this change; we choose other ingredients and create alternative fingerprints which are not sensitive to the particular change, but sensitive to others. Then, after making the model change, the fingerprint tests should pass, and the model should be verified. When the model is verified, the baseline ingredients can be restored.

Note

Fingerprint testing depends on repeatable deterministic simulations, random number generation, etc. If the tests are not repeatable, i.e. they don’t result in the same fingerprints when run multiple times (e.g. emulation), fingerprint tests are useless.