Multiplayer Desync Audit Trail

When a client and server disagree on the value of a networked variable, a desync occurs. Desyncs can be extremely difficult to debug which is where the Multiplayer Desync Audit Trail can help.

The Multiplayer Desync Audit Trail is a tool available with the multiplayer gem for debug-configuration builds. It details all network desyncs that occur and categorizes network activity leading up to a desync, allowing you to root cause the desync itself.

The Audit Trail will capture a desync including the input ID and host frame it occurred on. For every desync, the UI will list all the captured activity leading to the desync. The depth of this history can be controlled by a cvar.

Auditing Categories

The Audit Trail tool is designed to capture desync events, but can also be configured to capture input events and custom events.

Desync

Desyncs are the primary event captured by the Audit Trail. Consequently the Audit Trail organizes all activity around desyncs. On the client, desyncs generally include a delta map of variables and the values the client and server disagreed on.

Input events

Network inputs detail actions that create deltas in the networked state of the simulation. The Audit Trail lists all network inputs that were sent. In addition it also lists non-default values for each member per network input. This allows the correlation of inputs to desynchronized data. By tracking client inputs and desyncs together, you can identify if player actions affect specific desyncs.

The above screenshot shows inputs that occurred on the client’s player entity, relative to the desync including those that are processed locally. For the host frames in question, the actions were exclusively the player moving via NetworkPlayerMovementComponent.

Capturing custom events

Custom events are custom auditing events the developer can specify via macros in their C++ source.

Custom events allow the developer to specify additional information they’d like to track. In order to use these macros, including 1MultiplayerDebug.h1 is required. The parameters the macros require are:

• The variable in question
• The variable’s underlying type. This should always be T, even for RewindableObject<T> values being captured.
• A NetworkInput for commands starting with AZ_MPAUDIT_INPUT. This binds the macro usage to a specific host frame and input ID, giving the ability to categorize by event time.

Multiplayer::RewindableObject<AZ::Vector3> NetworkExampleComponent::GetVelocity()
{
    return m_velocity;
}

void NetworkExampleComponent::OnExampleEvent(Multiplayer::NetworkInput& input)
{
    // Pass the input, the RewindableObject and its templated type of Vector3
    AZ_MPAUDIT_INPUT_REWINDABLE(input, GetVelocity(), AZ::Vector3);
}

Configuration

Various cvars can be modified to enable additional data capture. Enabling additional capture both adds a performance cost and density to the Audit Trail. If you require capturing a longer event history, consider increasing the value of net_DebutAuditTrail_HistorySize.