Pac-Mind++
Ghost from the right- Seeking
- Caution
- Seeking again
The agent detours around the ghost, then resumes collecting pellets.
HazardRooms
Spikes in the lower passage- Seeking
- Caution
- Seeking again
The agent avoids the spikes, unlocks the door, then continues to the amulet.
A trained policy can favor reward or caution, depending on a control input. A calibrated risk estimate guides when to switch between them. The policy's network weights stay fixed.
Caution and progress
A useful cautious behavior can prevent a failure and still cost reward or time. Leaving it active indefinitely can turn a detour into waiting. The deployment rule has to decide both when to enter caution and when to release it.
The policy supplies a range of behaviors. A supervisor chooses among them using a risk estimate calibrated on validation rollouts. Keeping the policy fixed lets us test the choice of behavior separately from the rule that selects it.
What changes at an alarm
From the same saved alarm states, we compared continuing to seek reward with entering caution. The cautious response reduced failure over the next 100 actions and improved eventual completion. Each continuation kept the original task deadline.
Pac-Mind++
- Continue seeking
- 12.6%
- Enter caution
- 5.0%
HazardRooms
- Continue seeking
- 18.4%
- Enter caution
- 7.9%
Failure within 100 actions. Paired continuations from common alarm states, with the same frozen actor. Caution ramps the cost weight from 0.2 to 0.7 over ten actions.
All settings and 95% intervals
| Intervention | Pac-Mind++ | HazardRooms |
|---|---|---|
| Continue seeking | 12.6 [11.4, 13.9] | 18.4 [16.7, 20.1] |
| Entropy-matched seeking | 8.4 [7.4, 9.5] | 13.1 [11.7, 14.6] |
| Cost weight to 0.7 | 5.0 [4.3, 5.8] | 7.9 [6.7, 9.2] |
| Cost weight to 0.8 | 3.9 [3.2, 4.7] | 6.2 [5.1, 7.5] |
More caution can also stall progress. Raising the response weight from 0.7 to 0.8 lowers short-window failure further, but Pac-Mind++ completion falls from 63.4% to 49.2%. Timeouts rise from 23.0% to 40.6%.
Releasing caution helps recover progress. In full Pac-Mind++ episodes from reset, completion rises from 62.1% to 64.6% compared with keeping caution active after the first alarm. Failure rises from 4.7% to 5.0%.
One policy across settings
In these tasks, sharing the actor across settings works better than dividing the same training interactions among eight specialists.
MC-PPO retains 98.0% and 97.2% of the full eight-actor action mixture's mean reward-cost coverage in the two tasks, using one-eighth of its policy-training interactions. Calibration rollouts are separate from that budget. CCPO remains a competitive single-policy alternative.
Figure 4. Test-set envelopes of the best observed reward within each cost ceiling. Both libraries shown here use logit mixing.
How the controls work
The policy is trained with two runtime inputs. A cost weight changes its action preferences. Temperature changes how concentrated its sampling is, without reordering the actions at a fixed observation and cost weight.
Learning the trade-off
Training uses separate reward and cost advantages, each measured against its critic's baseline. PPO combines them using the cost weight \(w\):
A larger \(w\) gives cost avoidance more influence during training. After training, \(w\) remains an input to the actor. Changing it selects a behavior without updating the network.
View figure ↗
When to change priorities
The supervisor estimates the chance of failure within 100 actions if the agent keeps seeking. The reversible version uses two thresholds:
- Enter caution when estimated risk rises above 10%.
- Return to seeking after five consecutive eligible decisions with estimated risk below 4%.
The weight moves between 0.2 and 0.7 over ten actions. Each mode lasts at least 20 decisions. Threshold counters stay off during ramps, and the release counter starts after that minimum duration. The gap between thresholds and the minimum duration prevent rapid switching.
Architecture and temperature
The cost weight \(w\) conditions two actor layers through FiLM. The combined advantage is normalized once per rollout for PPO; the reward and cost value losses remain separate.
Temperature acts only on the final logits. At a fixed observation and weight, increasing temperature preserves the action ranking and cannot decrease entropy.
The supervisor evaluates the cost critic at a fixed seeking setting throughout the episode. Isotonic calibration maps that score to the probability of failure within 100 actions under continued seeking. Keeping the reference setting fixed prevents a control change from lowering the score just by changing the critic's input.
Calibrating a cost request
A cost weight is not a budget in task units. Validation selects the highest-reward setting whose upper estimate of mean episode cost passes the request \(b\):
The selected setting is held fixed on unseen layouts. At \(b=0.20\), mean cost is 0.168 [0.158, 0.178] in Pac-Mind++ and 0.167 [0.157, 0.177] in HazardRooms. The aggregate means pass the request, but estimated layout-level means exceed it in 18.8% and 19.8% of seed-layout cells.
Increasing ghost pursuit from 75% to 90% raises Pac-Mind++ cost at the same request to 0.248 under the original calibration. Recalibrating with shifted validation rollouts brings it to 0.181 without changing the actor, while reward in the shifted task falls from 134.2 to 128.3.
Looking ahead
Training gives an agent its capabilities. Deployment brings changing priorities, deadlines and risks. This work asks how much of that change a trained policy can handle through its inputs, and how to judge when a response is useful.
The navigation tasks are a starting point. The same question applies wherever an agent needs to adapt while it is already at work. Extending the approach will mean testing new controls, checking their behavior as conditions change, and identifying when further training is necessary.
Evaluation limits and calibration costs
The evaluation covers discrete, partially observed navigation with explicit hazard costs. Increasing the cost weight can raise total episode cost even when discounted cost falls, so deployment selection uses measured outcomes. The risk audit also finds underprediction in states visited during caution and after release.
Calibration provides a common probability scale; separately tuned raw critic thresholds perform similarly. Static calibration requires 28,928 episodes per trained replicate, and alarm fitting and checking add 1,024 seeking episodes. Those costs are outside the policy-training comparison.