Best Practices and Troubleshooting
Best Practices
Campaign Configuration
Start with Basic Strategies: Begin with "basic" and "jailbreak" strategies for comprehensive baseline coverage.
Select Relevant Probes: Choose probe types that match your application's risk profile.
Provide Detailed Context: Include comprehensive application purpose descriptions for more accurate testing.
Iterative Testing: Run multiple campaigns as you implement mitigations.
Results Analysis
Prioritize by Severity: Address Critical and High severity vulnerabilities first.
Review Failed Tests: Examine actual attack conversations to understand exploitation techniques.
Track Progress: Compare campaigns over time to measure security improvements.
Document Mitigations: Record what defenses were implemented for each vulnerability.
Security Posture Management
Regular Testing: Schedule periodic adversary campaigns.
Comprehensive Coverage: Test all probe types relevant to your use case.
Defense Validation: Re-run campaigns after implementing security controls.
Trend Analysis: Monitor your pass rate and vulnerability counts over time.
Technical Architecture Notes
Probe Types
Probes are modular test components that can be:
Combined with multiple strategies
Executed in parallel
Customized with application-specific context
Strategy Execution
Strategies modify probe prompts through:
Direct transformation (basic)
Obfuscation techniques (encoding)
Multi-turn conversations (advanced)
Systematic search (tree-based)
Troubleshooting
Common Issues
Campaign Fails to Start:
Verify target credentials are correct.
Check API endpoint connectivity.
Ensure deployment name matches Azure configuration.
Low Pass Rates:
Review application purpose and system prompts.
Implement additional input validation.
Add content filtering layers.
Consider RBAC controls.
Inconsistent Results:
Model responses may vary between runs.
Temperature and sampling settings affect consistency.
Run multiple campaigns for statistical significance.