Port Following Behavior When Real Server is Down


I'm trying to understand if there is a way around the behavior I'm seeing when using port following rules. Here's what I'm starting with:

3 virtual services: one "main" VS for HTTPS and two "secondary" VSes for different UDP ports. The two VSes for the UDP ports refer back to the main rule for HTTPS for port following, such that whichever real server is selected for the HTTPS rule will be the server they are directed to for both the other UDP VSes. Affinity needs to be maintained for the UDP ports at all times; if a session is directed to another real server for the HTTPS rule, it must also be redirected to the same real server for either of the two UDP rules. It cannot persist to the original real server.

What I'm experiencing is this: if there is an issue with the HTTPS service only on a real server that the client is connected to, they are properly directed to a new real server, however the subsequent UDP traffic does not follow to the new real server and is instead persisted to the original real server, thus breaking the affinity that is required by the protocol.

Is this normal behavior? Does session persistence take precedence over port following?