Effects of Time Synchronization on Time-Aware Shaping¶

Goals¶

In this showcase we demonstrate how time synchronization affects end-to-end delay in a network that is using time-aware traffic shaping.

Note

This showcase builds upon the Clock Drift, Using gPTP and Time-Aware Shaping showcases.

INET version: 4.4

Source files location: inet/showcases/tsn/combiningfeatures/gptpandtas

The Model¶

Overview¶

The showcase contains a network that uses time-aware shaping and inaccurate local clocks in each network node. Time-aware shaping requires synchronized time among all network nodes, so we also use gPTP for time synchronization. This results in a constant end-to-end delay for the traffic. We examine how failure and recovery in time synchronization affect the delay.

If time synchronization fails for some reason, e.g. due to the primary master clock going offline, the end-to-end delay guarantees of time-aware shaping cannot be met anymore. However, time synchronization can continue and delay guarantees can be met if all network nodes switch over to a secondary master clock.

To demonstrate this, we present three cases, with three configurations:

Normal operation: time synchronization works, and the delay is constant (this is the same case as in last configuration in the Using gPTP showcase).
Failure of master clock: the master clock disconnects from the network, and time is not synchronized anymore
Failover to a secondary master clock: the master clock disconnects from the network, but time synchronization can continue because network nodes switch to the secondary master clock

The Configuration¶

All simulations in the showcase use the same network as in the sh:tsn:timesync:gptp:redundancy section of the Using gPTP showcase. It constains TsnDevice and TsnClock modules connected to a ring of switches (TsnSwitch):

Most settings, such as traffic generation, time-aware shaping, clock drift, gPTP domains, and visualization, are shared between the three cases, and are specified in the General configuration. The following sections present these settings.

Traffic¶

Traffic in the network consists of UDP packets sent between tsnDevice1 and tsnDevice4, and gPTP messages. sent by all nodes. To generate the UDP application traffic, we configure tsnDevice1 to send 10B UDP packets to tsnDevice2:

*.tsnDevice1.numApps = 1
*.tsnDevice1.app[0].typename = "UdpSourceApp"
*.tsnDevice1.app[0].source.packetLength = 10B
*.tsnDevice1.app[0].source.productionInterval = 1ms
*.tsnDevice1.app[0].io.destAddress = "tsnDevice4"
*.tsnDevice1.app[0].io.destPort = 1000
*.tsnDevice1.app[0].source.clockModule = "^.^.clock"

Clocks and Clock Drift¶

Clocks are configured similarly to the TODO showcase, using MultiClock and SettableClock modules with constant drift oscillators. However, we configure the TODO oscillator a bit differently to make the failover effect in the third example simulation more visible:

*.tsnClock2.clock.clock[*].oscillator.driftRate = -30ppm
**.oscillator.driftRate = uniform(-100ppm, 100ppm)

We specify a random drift rate for all oscillators in the second line, but override this setting for the oscillators of TsnClock2 in the first line with a specific constant drift rate, in order to make the failover effect more visible.

TODO minden time domainnek a timeja minden nodeban nyilvan van tartva -> ez a multiclock lenyege

Time-Aware Shaping¶

Here is the time-aware shaping part of the configuration:

*.tsnSwitch*.hasEgressTrafficShaping = true
*.tsnSwitch*.eth[*].macLayer.queue.numTrafficClasses = 2
*.tsnSwitch*.eth[*].macLayer.queue.classifier.typename = "ContentBasedClassifier"
*.tsnSwitch*.eth[*].macLayer.queue.queue[*].typename = "DropTailQueue"
*.tsnSwitch*.eth[*].macLayer.queue.classifier.packetFilters = ["app*", "Gptp*"]
*.tsnSwitch*.eth[*].macLayer.queue.transmissionGate[0].durations = [100us, 900us]
*.tsnSwitch*.eth[*].macLayer.queue.transmissionGate[0].clockModule = "^.^.^.^.clock"
*.tsnSwitch*.eth[*].macLayer.queue.transmissionGate[1].initiallyOpen = true
*.tsnSwitch*.eth[*].macLayer.queue.transmissionGate[1].durations = []		# gate always open
*.tsnSwitch*.eth[*].macLayer.queue.transmissionGate[1].clockModule = "^.^.^.^.clock"

We enable time-aware shaping in all switches. We configure two traffic classes: the higher priority class for gPTP messages, and the lower priority class for UDP packets. The gPTP messages are critical to maintain time synchronization in the network, so we assign them higher priority, and configure the time-aware shapers to not schedule gPTP messages, but forward them immediatelly (the corresponding gates are always open). Gates for the UDP packets are configured to be open 10% of the time, so the UDP traffic is limited to 10Mbps. The UDP traffic data rate is 10kbps, so there is a lot of room in the send window; this makes the setup more tolerant of clock drift between network nodes.

gPTP¶

For the time synchronization, we use the same clock and gPTP setup as in the Two Master Clocks Exploiting Network Redundancy configuration in the Using gPTP showcase. To summarize, we have a primary and a hot-standby master clock node (tsnClock1 and tsnClock2, respectively). Each master clock has two gPTP time domains, and each gPTP domain disseminates timing information in the ring of switches in opposing directions. All bridge and slave nodes are part of all four time domains. With this setup, time synchronization is protected against the failure of one master clock, and the failure of any link between the switches. The two gPTP domains of the hot-standby master clock synchronize to the two domains of the primary master. The switches and the devices have four gPTP domains, and four up-to-date clocks during normal operation.

Note

Here is the gPTP configuration for reference:

click arrow to open/close

*.tsnClock2.clock.typename = "MultiClock"
*.tsnClock2.clock.numClocks = 2

# TSN clocks have multiple gPTP time synchronization domains
*.tsnClock*.gptp.typename = "MultiDomainGptp"
*.tsnClock1.gptp.numDomains = 2
*.tsnClock1.gptp.domain[0..1].clockModule = "tsnClock1.clock"
*.tsnClock1.gptp.domain[0].masterPorts = ["eth0"]
*.tsnClock1.gptp.domain[1].masterPorts = ["eth0"]
*.tsnClock2.gptp.numDomains = 4
*.tsnClock2.gptp.domain[2..3].clockModule = "tsnClock2.clock"
*.tsnClock2.gptp.domain[0].gptpNodeType = "SLAVE_NODE"
*.tsnClock2.gptp.domain[0].slavePort = "eth0"
*.tsnClock2.gptp.domain[1].gptpNodeType = "SLAVE_NODE"
*.tsnClock2.gptp.domain[1].slavePort = "eth0"
*.tsnClock2.gptp.domain[2].gptpNodeType = "MASTER_NODE"
*.tsnClock2.gptp.domain[2].masterPorts = ["eth0"]
*.tsnClock2.gptp.domain[3].gptpNodeType = "MASTER_NODE"
*.tsnClock2.gptp.domain[3].masterPorts = ["eth0"]

# TSN switches have multiple clocks
*.tsnSwitch*.clock.typename = "MultiClock"
*.tsnSwitch*.clock.numClocks = 4

# TSN switches have multiple gPTP time synchronization domains
*.tsnSwitch*.gptp.typename = "MultiDomainGptp"
*.tsnSwitch*.gptp.numDomains = 4

# TSN switch 1
*.tsnSwitch1.gptp.domain[0].masterPorts = ["eth1"]
*.tsnSwitch1.gptp.domain[0].slavePort = "eth0"
*.tsnSwitch1.gptp.domain[1].masterPorts = ["eth2"]
*.tsnSwitch1.gptp.domain[1].slavePort = "eth0"
*.tsnSwitch1.gptp.domain[2].masterPorts = ["eth1"]
*.tsnSwitch1.gptp.domain[2].slavePort = "eth2"
*.tsnSwitch1.gptp.domain[3].masterPorts = ["eth2"]
*.tsnSwitch1.gptp.domain[3].slavePort = "eth1"

# TSN switch 2
*.tsnSwitch2.gptp.domain[0].masterPorts = ["eth1", "eth2"]
*.tsnSwitch2.gptp.domain[0].slavePort = "eth0"
*.tsnSwitch2.gptp.domain[1].masterPorts = ["eth2"]
*.tsnSwitch2.gptp.domain[1].slavePort = "eth1"
*.tsnSwitch2.gptp.domain[2].masterPorts = ["eth1", "eth2"]
*.tsnSwitch2.gptp.domain[2].slavePort = "eth0"
*.tsnSwitch2.gptp.domain[3].masterPorts = ["eth0", "eth2"]
*.tsnSwitch2.gptp.domain[3].slavePort = "eth1"

# TSN switch 3
*.tsnSwitch3.gptp.domain[0].masterPorts = ["eth1", "eth2"]
*.tsnSwitch3.gptp.domain[0].slavePort = "eth0"
*.tsnSwitch3.gptp.domain[1].masterPorts = ["eth0", "eth2"]
*.tsnSwitch3.gptp.domain[1].slavePort = "eth1"
*.tsnSwitch3.gptp.domain[2].masterPorts = ["eth2"]
*.tsnSwitch3.gptp.domain[2].slavePort = "eth0"
*.tsnSwitch3.gptp.domain[3].masterPorts = ["eth0", "eth2"]
*.tsnSwitch3.gptp.domain[3].slavePort = "eth1"

# TSN switch 4
*.tsnSwitch4.gptp.domain[0].masterPorts = ["eth0", "eth2"]
*.tsnSwitch4.gptp.domain[0].slavePort = "eth1"
*.tsnSwitch4.gptp.domain[1].masterPorts = ["eth0", "eth1"]
*.tsnSwitch4.gptp.domain[1].slavePort = "eth2"
*.tsnSwitch4.gptp.domain[2].masterPorts = ["eth2"]
*.tsnSwitch4.gptp.domain[2].slavePort = "eth0"
*.tsnSwitch4.gptp.domain[3].masterPorts = ["eth1"]
*.tsnSwitch4.gptp.domain[3].slavePort = "eth0"

# TSN switch 5
*.tsnSwitch5.gptp.domain[0].masterPorts = ["eth1", "eth2"]
*.tsnSwitch5.gptp.domain[0].slavePort = "eth0"
*.tsnSwitch5.gptp.domain[1].masterPorts = ["eth0", "eth2"]
*.tsnSwitch5.gptp.domain[1].slavePort = "eth1"
*.tsnSwitch5.gptp.domain[2].masterPorts = ["eth1", "eth2"]
*.tsnSwitch5.gptp.domain[2].slavePort = "eth0"
*.tsnSwitch5.gptp.domain[3].masterPorts = ["eth2"]
*.tsnSwitch5.gptp.domain[3].slavePort = "eth1"

# TSN switch 6
*.tsnSwitch6.gptp.domain[0].masterPorts = ["eth2"]
*.tsnSwitch6.gptp.domain[0].slavePort = "eth0"
*.tsnSwitch6.gptp.domain[1].masterPorts = ["eth0", "eth2"]
*.tsnSwitch6.gptp.domain[1].slavePort = "eth1"
*.tsnSwitch6.gptp.domain[2].masterPorts = ["eth1", "eth2"]
*.tsnSwitch6.gptp.domain[2].slavePort = "eth0"
*.tsnSwitch6.gptp.domain[3].masterPorts = ["eth0", "eth2"]
*.tsnSwitch6.gptp.domain[3].slavePort = "eth1"

# TSN devices have multiple clocks
*.tsnDevice*.clock.typename = "MultiClock"
*.tsnDevice*.clock.numClocks = 4

# TSN devices have multiple gPTP time synchronization domains
*.tsnDevice*.gptp.typename = "MultiDomainGptp"
*.tsnDevice*.gptp.numDomains = 4
*.tsnDevice1.gptp.clockModule = "tsnDevice1.clock"
*.tsnDevice2.gptp.clockModule = "tsnDevice2.clock"
*.tsnDevice3.gptp.clockModule = "tsnDevice3.clock"
*.tsnDevice4.gptp.clockModule = "tsnDevice4.clock"
*.tsnDevice*.gptp.domain[*].slavePort = "eth0"

# different initial gPTP pdelay measurement and time synchronization offsets
**.pdelayInitialOffset = 0.1ms
*.*.gptp.domain[0].syncInitialOffset = syncInterval * 1 / 4
*.*.gptp.domain[1].syncInitialOffset = syncInterval * 2 / 4
*.*.gptp.domain[2].syncInitialOffset = syncInterval * 3 / 4
*.*.gptp.domain[3].syncInitialOffset = syncInterval * 4 / 4

The Example Simulations and Results¶

Normal Operation¶

In this configuration, time synchronization is operating properly. The configuration for this case is empty (only the General configuration applies):

[Config NormalOperation]
description = "Normal operation of time-aware shaping with time synchronization"

Let us examine the clock drift (i.e. clock time difference to simulation time) and the end-to-end delay. The following chart displays the clock drift for clocks in gPTP time domain 0 (one of the gPTP domains of the primary master). This is the active gPTP time domain in all network nodes (i.e. all nodes keep time according to this gPTP time domain):

../../../../../_images/NormalOperation_domain0.png

Due to everything in the network operating properly, bridge and slave nodes periodically synchronize their clocks to the primary master node.

Note

The other three gPTP time domains are maintained simultaneously, but not used.

The next chart displays the clock drift for gPTP time domain 2, where timing information originates from the hot-standby master clock. All bridge and slave nodes synchronize to the hot-standby master (thick orange line), which in turn synchronizes to the primary master (dotted blue line) in another gPTP domain:

Note

In this configuration, this time domain is not used by applications or the time-aware shaping. Later on, this will be important because clocks will fail over to this gPTP time domain.

../../../../../_images/NormalOperation_domain2.png

Note that bridge and slave nodes update their time when the hot-standby master node’s clock has already drifted from the primary master somewhat.

The next chart displays the end-to-end delay of application traffic, which is a constant low value:

../../../../../_images/delay_normaloperation.png

In the next section, we examine what happens if we take the master clock node offline during the simulation.

Link Failure of Master Clock¶

In this configuration, the time synchronization of the two time domains of the primary master clock stop due to a link failure of the primary master clock (the other two time domains are still synchronized). After the link break, the active clocks in the network begin to diverge from each other.

We schedule the link break at 2s (halfway through the simulation) with a scenario manager script. Here is the configuration:

[Config LinkFailure]
description = "Master clock link failure effects on time-aware shaping"
extends = NormalOperation

# TSN clock1 disconnects from the network at 2 seconds
*.scenarioManager.script = xml("<scenario> \
                                  <at t='2'> \
                                    <disconnect src-module='tsnClock1' src-gate='ethg[0]'/> \
                                  </at> \
                                </scenario>")

As in the previous section, we take a look at clock drift and end-to-end delay. The next chart shows the clock drift of gPTP time domain 0:

../../../../../_images/LinkFailure_domain0.png

The clocks begin to diverge after time synchronization in this gPTP time domain stops.

TODO domain -> time domain! too generic

The next chart shows clock drift in domain 2. This is the domain of the hot-standby master clock, which stays online. Thus the clocks in this domain keep being synchronized. (the hot-standby master is denoted with the thick orange line, the primary master with the dotted blue line.)

TODO keep being synchronized

../../../../../_images/LinkFailure_domain2.png

Note that when the primary master node goes offline, the hot-standby master node cannot synchronize to it any more. Thus its clock drifts from the primary master’s, denoted by the orange and blue lines diverging. The bridge and slave nodes continue to synchronize to the hot-standby master node. (denoted by the other lines following it)

The next chart shows the active clocks in bridge and slave nodes:

../../../../../_images/LinkFailure_endstations.png

The active clocks are still the ones using domain 0 as the source of time, thus their times diverge after the link break TODO due to not being synced. Again, this chart is the same as the one displaying domain 0.

TODO nem kell a chart csak annyi hogy ugynaaz

The next chart shows the delay:

../../../../../_images/delay_linkfailure.png

After the clock divergence grows above a certain value the end-to-end delay suddenly increases dramatically. The reason is that frames often wait for the next gate scheduling cycle because they miss the allocated time slot due to improperly synchronized clocks. The delay increases from the nominal microsecond-range to millisecond range. Needless to say, in this scenario, the constant low delay cannot be guaranteed.

In the next section, we examine what happens if time synchronization can continue after the link break due to failover to the hot-standby master clock.

Failover to Hot-Standby Clock¶

In this configuration, we take the primary master clock offline just as in the previous one, but we also switch the active clock in each node over to the one that synchronizes to the hot-standby master, so time synchronization can continue to keep the difference of clocks in the network below the required limit.

We schedule the link break with a scenario manager script. We also schedule changing the active clock parameter in the MultiClock modules in all nodes. Both changes are scheduled at 2s, halfway through the simulation:

Note

We schedule the two changes at the same time, 2s. This is unrealistic.

[Config Failover]
description = "Hot-standby master clock takes over time synchronization"
extends = LinkFailure

# TSN clock1 disconnects from the network at 2 seconds
# all multi clocks switch to the 2nd clock submodule
*.scenarioManager.script = xml("<scenario> \
                                  <at t='2'> \
                                    <disconnect src-module='tsnClock1' src-gate='ethg[0]'/> \
                                  </at> \
                                  <at t='2'> \
                                    <set-param module='tsnSwitch1.clock' par='activeClockIndex' value='2'/> \
                                    <set-param module='tsnSwitch2.clock' par='activeClockIndex' value='2'/> \
                                    <set-param module='tsnSwitch3.clock' par='activeClockIndex' value='2'/> \
                                    <set-param module='tsnSwitch4.clock' par='activeClockIndex' value='2'/> \
                                    <set-param module='tsnSwitch5.clock' par='activeClockIndex' value='2'/> \
                                    <set-param module='tsnSwitch6.clock' par='activeClockIndex' value='2'/> \
                                    <set-param module='tsnDevice1.clock' par='activeClockIndex' value='2'/> \
                                    <set-param module='tsnDevice2.clock' par='activeClockIndex' value='2'/> \
                                    <set-param module='tsnDevice3.clock' par='activeClockIndex' value='2'/> \
                                    <set-param module='tsnDevice4.clock' par='activeClockIndex' value='2'/> \
                                  </at> \
                                </scenario>")

Here are the results:

Let’s examine the results. The clock drifts in domain 0 (clock time of the primary master node) are displayed on the following chart:

../../../../../_images/Failover_domain0.png

The clocks begin to diverge from each other after the link break.

The next chart displays the clock drifts in domain 2 (clock time of the hot-standby master node):

../../../../../_images/Failover_domain2.png

After the link break, the clocks are synchronized to the hot-standby master’s time.

The next chart displays the clock drift of the active clock in all nodes:

../../../../../_images/Failover_endstations.png

The following chart displays the delay:

The delay is constant and the same as during normal operation, due to the seamless failover to the hot-standby master node. For comparison, the next chart displays the delay for the three configurations on individual plots, with the same axis scale:

Sources: omnetpp.ini, GptpAndTasShowcase.ned

Discussion¶

Use this page in the GitHub issue tracker for commenting on this showcase.