Best Practices and Troubleshooting

Best Practices

Campaign Configuration

1. Start with Basic Strategies: Begin with "basic" and "jailbreak" strategies for comprehensive baseline coverage.

2. Select Relevant Probes: Choose probe types that match your application's risk profile.

3. Provide Detailed Context: Include comprehensive application purpose descriptions for more accurate testing.

4. Iterative Testing: Run multiple campaigns as you implement mitigations.

Results Analysis

1. Prioritize by Severity: Address Critical and High severity vulnerabilities first.

2. Review Failed Tests: Examine actual attack conversations to understand exploitation techniques.

3. Track Progress: Compare campaigns over time to measure security improvements.

4. Document Mitigations: Record what defenses were implemented for each vulnerability.

Security Posture Management

1. Regular Testing: Schedule periodic adversary campaigns.

2. Comprehensive Coverage: Test all probe types relevant to your use case.

3. Defense Validation: Re-run campaigns after implementing security controls.

4. Trend Analysis: Monitor your pass rate and vulnerability counts over time.

Technical Architecture Notes

Probe Types

Probes are modular test components that can be:

• Combined with multiple strategies

• Executed in parallel

• Customized with application-specific context

Strategy Execution

Strategies modify probe prompts through:

• Direct transformation (basic)

• Obfuscation techniques (encoding)

• Multi-turn conversations (advanced)

• Systematic search (tree-based)

Troubleshooting

Common Issues

• Campaign Fails to Start:

  • Verify target credentials are correct.

  • Check API endpoint connectivity.

  • Ensure deployment name matches Azure configuration.

• Low Pass Rates:

  • Review application purpose and system prompts.

  • Implement additional input validation.

  • Add content filtering layers.

  • Consider RBAC controls.

• Inconsistent Results:

  • Model responses may vary between runs.

  • Temperature and sampling settings affect consistency.

  • Run multiple campaigns for statistical significance.