1. Home
  2. Countermeasure
  3. Reinforcement Learning

Reinforcement Learning

A reinforcement learning (RL) agent deployed within the spacecraft or ground system can provide an adaptive, autonomous anomaly detection and response capability that identifies anomalous events, including malicious data inputs and injected commands, and redirects affected processes to proceed safely by ignoring or isolating the malicious input. An RL agent learns a response policy that maps observations to actions according to its training environment and reward function. It may generalize to scenarios not explicitly included in training, but its ability to detect or respond correctly to novel attacks or conditions outside the validated operational envelope must not be assumed. Anomaly detection may be incorporated into the RL architecture or provided by a separate monitoring function. Effective deployment requires separate protections against compromise of the training process and manipulation of observations presented to the deployed agent. Online learning or policy adaptation should be disabled unless specifically authorized, bounded, and validated. Agent-selected responses must be constrained by a trusted safety mechanism.

Sources

ID: CM0068
Tier: III
Onboard SV CM 
Created: 2022/10/19
Last Modified: 2026/08/06

Pre-Operations Government

Acquisition requirements should address reinforcement learning-based anomaly detection as a candidate autonomous security capability for missions where the threat environment and operational autonomy requirements justify the development and validation investment, with specifications defining the state variables the agent will observe, the action space available to the agent, the reward function governing agent behavior, and the safety constraints that bound permissible agent responses. Requirements should specify that the RL agent be trained, validated, and tested against representative nominal, off-nominal, and adversarial scenarios before operational deployment. Security evaluation must separately address poisoning of training or policy-update artifacts and runtime manipulation of observations presented to the deployed agent. Contract language should require that the RL agent architecture, training environment, training data provenance, and validation methodology be documented as controlled security engineering deliverables subject to government technical review. Evaluation criteria should assess offerors' proposed RL architecture, their training scenario coverage of the mission's threat environment, their approach to constraining agent actions to a safe response set, and their demonstrated experience deploying RL-based security functions in safety- and mission-critical embedded systems. Verification should include held-out off-nominal and adversarial scenarios that were not used to train the policy. Testing must measure detection and response performance against defined acceptance criteria and verify that the independent safety mechanism prevents prohibited or unsafe responses. Successful testing demonstrates performance only within the evaluated conditions and must not be interpreted as proof that the agent will respond correctly to all novel attacks.

Pre-Operations Developer/Supplier

Reinforcement learning agent design must begin with a formal definition of the observation space, action space, and reward function that governs the agent's behavior, as these three elements determine what the agent will learn and what responses it will select; poorly defined reward functions or unconstrained action spaces can produce agent behavior that is effective against training scenarios but harmful or exploitable in operational deployment. The approved action authority must be enforced by a trusted runtime mechanism outside the RL policy. The mechanism must reject agent outputs that violate defined safety constraints and invoke an approved fallback response when necessary. Constraining the policy’s output layer alone is insufficient because an individually permitted action may still be unsafe in the current state or sequence of operations. Operational policy adaptation must not expand the agent’s approved action authority. Training environments must faithfully represent the spacecraft's operational state dynamics, the expected range of nominal inputs, and a diverse set of adversarial input scenarios, with the adversarial scenarios developed from the mission's threat model to ensure that the agent's learned policy is relevant to the actual attack surface. Observation inputs supplied to the RL agent must be protected against unauthorized modification and evaluated for validity before use. Integrity verification and input validation can detect some forms of corruption but do not establish that an authenticated or syntactically valid observation accurately represents the spacecraft state. The design must address adversarially crafted or compromised-source observations within the mission threat model. Computational resource requirements for RL agent inference must be characterized and verified to be within the flight platform's processing and power budget without degrading the primary mission functions the agent is designed to protect.