itqop
/
brief-rags-bench
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
brief-rags-bench/tests/e2e/README.md

438 lines
11 KiB
Markdown
Raw Blame History

# End-to-End (E2E) Tests
End-to-end tests for the complete Brief Bench FastAPI system, testing the entire stack from authentication through RAG queries to data persistence.
## Overview
E2E tests validate:
- Complete user workflows from login to query results
- Integration between FastAPI backend, DB API, and RAG backends
- Real API calls to all external services (no mocking)
- Data persistence and retrieval
- Cross-environment functionality (IFT, PSI, PROD)
- Error handling and edge cases
## Prerequisites
**CRITICAL**: All external services must be running before E2E tests can execute.
### Required Services
1. **DB API** (database service)
- Must be running and accessible
- Default: `http://localhost:8081`
- Health check: `GET http://localhost:8081/health`
2. **RAG Backends** (one or more environments)
- **IFT RAG**: Development/test environment
- **PSI RAG**: Backend mode testing
- **PROD RAG**: Production-like testing
- Each environment needs its own RAG backend server running
3. **Test User Account**
- A valid 8-digit test login must exist in DB API
- Recommended: Use a dedicated test account (e.g., `99999999`)
- This account will be used for all E2E test operations
### Service Availability Check
E2E tests automatically check prerequisites before running:
- DB API health endpoint
- RAG backend host configurations
- Test credentials presence
If any prerequisite is not met, tests will be **skipped** with a detailed error message.
## Environment Configuration
### Create `.env.e2e` File
Copy the example file and configure for your environment:
```bash
cp tests/e2e/.env.e2e.example tests/e2e/.env.e2e
```
### Required Environment Variables
Edit `tests/e2e/.env.e2e` with your configuration:
```bash
# DB API Configuration
E2E_DB_API_URL=http://localhost:8081/api/v1
# Test User Credentials
E2E_TEST_LOGIN=99999999 # 8-digit test user login
# IFT Environment Settings (Bench Mode)
E2E_IFT_RAG_HOST=ift-rag.example.com
E2E_IFT_BEARER_TOKEN=your_ift_bearer_token_here
E2E_IFT_SYSTEM_PLATFORM=telegram
E2E_IFT_SYSTEM_PLATFORM_USER=test_user_ift
# PSI Environment Settings (Backend Mode)
E2E_PSI_RAG_HOST=psi-rag.example.com
E2E_PSI_BEARER_TOKEN=your_psi_bearer_token_here
E2E_PSI_PLATFORM_USER_ID=test_user_psi
E2E_PSI_PLATFORM_ID=telegram
# PROD Environment Settings (Bench Mode)
E2E_PROD_RAG_HOST=prod-rag.example.com
E2E_PROD_BEARER_TOKEN=your_prod_bearer_token_here
E2E_PROD_SYSTEM_PLATFORM=telegram
E2E_PROD_SYSTEM_PLATFORM_USER=test_user_prod
```
### Security Note
⚠️ **NEVER commit `.env.e2e` to version control!**
The `.env.e2e` file contains:
- Real bearer tokens for RAG backends
- Production-like credentials
- Sensitive configuration
Always use `.env.e2e.example` as a template.
## Running E2E Tests
### Prerequisites Check
First, ensure all services are running:
```bash
# Check DB API
curl http://localhost:8081/health
# Check that your .env.e2e is configured
cat tests/e2e/.env.e2e
```
### Run All E2E Tests
```bash
# Activate virtual environment
.venv\Scripts\activate # Windows
source .venv/bin/activate # Linux/Mac
# Run all E2E tests
pytest tests/e2e/ -v -m e2e
# Run with detailed output
pytest tests/e2e/ -v -m e2e -s
```
### Run Specific Test Categories
```bash
# Run only IFT environment tests
pytest tests/e2e/ -v -m e2e_ift
# Run only PSI environment tests
pytest tests/e2e/ -v -m e2e_psi
# Run only PROD environment tests
pytest tests/e2e/ -v -m e2e_prod
# Run only workflow tests
pytest tests/e2e/test_full_flow_e2e.py -v
# Run only error scenario tests
pytest tests/e2e/test_error_scenarios_e2e.py -v
# Run only RAG backend tests
pytest tests/e2e/test_rag_backends_e2e.py -v
```
### Run Individual Test
```bash
# Run a specific test function
pytest tests/e2e/test_full_flow_e2e.py::TestCompleteUserFlow::test_full_workflow_bench_mode -v
```
### Useful pytest Options
```bash
# Show print statements
pytest tests/e2e/ -v -s
# Stop on first failure
pytest tests/e2e/ -v -x
# Show local variables on failure
pytest tests/e2e/ -v -l
# Run with coverage (not typical for E2E)
pytest tests/e2e/ -v --cov=app
# Increase timeout for slow RAG backends
pytest tests/e2e/ -v --timeout=300
```
## Test Structure
### Test Files
```
tests/e2e/
├── conftest.py # E2E fixtures and configuration
├── .env.e2e.example # Example environment variables
├── .env.e2e # Your actual config (not in git)
├── README.md # This file
├── test_full_flow_e2e.py # Complete user workflow tests
├── test_rag_backends_e2e.py # RAG backend integration tests
└── test_error_scenarios_e2e.py # Error handling and edge cases
```
### Test Markers
E2E tests use pytest markers for categorization:
- `@pytest.mark.e2e` - All E2E tests
- `@pytest.mark.e2e_ift` - IFT environment specific
- `@pytest.mark.e2e_psi` - PSI environment specific
- `@pytest.mark.e2e_prod` - PROD environment specific
### Fixtures
Key fixtures from `conftest.py`:
- `check_prerequisites` - Verifies all services are available
- `e2e_client` - FastAPI TestClient instance
- `e2e_auth_headers` - Authenticated headers with JWT token
- `setup_test_settings` - Configures user settings for all environments
- `cleanup_test_sessions` - Removes test data after each test
## Test Coverage
### Complete User Workflows (`test_full_flow_e2e.py`)
1. **Full Workflow - Bench Mode**
- Authenticate → Get settings → Send bench query → Save session → Retrieve → Delete
2. **Full Workflow - Backend Mode**
- Authenticate → Verify PSI settings → Send backend query → Save session
3. **Settings Change Affects Queries**
- Change settings → Verify mode compatibility → Restore settings
4. **Multiple Sessions Management**
- Create sessions for all environments → List → Filter → Delete all
5. **User Data Isolation**
- Verify authentication requirements → Test access controls
### RAG Backend Tests (`test_rag_backends_e2e.py`)
1. **Environment-Specific Queries**
- IFT bench mode queries
- PSI backend mode queries
- PROD bench mode queries
2. **Backend Mode Features**
- Session management
- Session reset functionality
- Sequential question processing
3. **Query Parameters**
- `with_docs` parameter handling
- Multiple questions in one request
- Cross-environment queries
### Error Scenarios (`test_error_scenarios_e2e.py`)
1. **Authentication Errors**
- Missing auth token
- Invalid JWT token
- Malformed authorization header
2. **Validation Errors**
- Invalid environment names
- Empty questions list
- Missing required fields
- Invalid data structures
3. **Mode Compatibility**
- Bench query with backend mode settings
- Backend query with bench mode settings
4. **Resource Not Found**
- Nonexistent session IDs
- Invalid UUID formats
5. **Settings Errors**
- Invalid API modes
- Invalid environments
6. **Edge Cases**
- Very long questions
- Special characters
- Large number of questions
- Pagination edge cases
## Timeouts
E2E tests use generous timeouts due to real RAG backend processing:
- **Default query timeout**: 120 seconds (2 minutes)
- **Large batch queries**: 180 seconds (3 minutes)
- **DB API operations**: 30 seconds
If tests timeout frequently, check:
1. RAG backend performance
2. Network connectivity
3. Server load
## Cleanup
Tests automatically clean up after themselves:
- `cleanup_test_sessions` fixture removes all sessions created during tests
- Each test is isolated and doesn't affect other tests
- Failed tests may leave orphaned sessions (check manually if needed)
### Manual Cleanup
If needed, clean up test data manually:
```python
# Get all sessions for test user
GET /api/v1/analysis/sessions?limit=1000
# Delete specific session
DELETE /api/v1/analysis/sessions/{session_id}
```
## Troubleshooting
### Tests Are Skipped
**Symptom**: All tests show as "SKIPPED"
**Causes**:
1. DB API not running
2. RAG backends not configured
3. Missing `.env.e2e` file
4. Test user doesn't exist
**Solution**: Check prerequisite error messages for details.
### Authentication Failures
**Symptom**: Tests fail with 401 Unauthorized
**Causes**:
1. Test user doesn't exist in DB API
2. Invalid test login in `.env.e2e`
3. JWT secret mismatch between environments
**Solution**: Verify test user exists and credentials are correct.
### Timeout Errors
**Symptom**: Tests timeout during RAG queries
**Causes**:
1. RAG backend is slow or overloaded
2. Network issues
3. Invalid bearer tokens
4. mTLS certificate problems
**Solution**:
- Check RAG backend health
- Verify bearer tokens are valid
- Increase timeout values if needed
### Connection Refused
**Symptom**: Connection errors to services
**Causes**:
1. Service not running
2. Wrong host/port in configuration
3. Firewall blocking connections
**Solution**: Verify all services are accessible and configuration is correct.
### Validation Errors (422)
**Symptom**: Tests fail with 422 Unprocessable Entity
**Causes**:
1. Incorrect data format in `.env.e2e`
2. Missing required settings
3. Invalid enum values
**Solution**: Check `.env.e2e.example` for correct format.
## Best Practices
### When to Run E2E Tests
- **Before deploying**: Always run E2E tests before production deployment
- **After major changes**: Run when modifying API endpoints or services
- **Regularly in CI/CD**: Set up automated E2E testing in your pipeline
- **Not during development**: Use unit/integration tests for rapid feedback
### Test Data Management
- Use dedicated test user account (not production users)
- Tests create and delete their own data
- Don't rely on existing data in DB
- Clean up manually if tests fail catastrophically
### Performance Considerations
- E2E tests are slow (real network calls)
- Run unit/integration tests first
- Consider running E2E tests in parallel (with caution)
- Use environment-specific markers to run subset of tests
### CI/CD Integration
Example GitHub Actions workflow:
```yaml
e2e-tests:
runs-on: ubuntu-latest
services:
db-api:
image: your-db-api:latest
ports:
- 8081:8081
steps:
- uses: actions/checkout@v3
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: pip install -r requirements.txt
- name: Create .env.e2e
run: |
echo "E2E_DB_API_URL=${{ secrets.E2E_DB_API_URL }}" > tests/e2e/.env.e2e
echo "E2E_TEST_LOGIN=${{ secrets.E2E_TEST_LOGIN }}" >> tests/e2e/.env.e2e
# ... other env vars
- name: Run E2E tests
run: pytest tests/e2e/ -v -m e2e
```
## Contributing
When adding new E2E tests:
1. Add test to appropriate file (workflow/backend/error)
2. Use existing fixtures from `conftest.py`
3. Add cleanup logic if creating persistent data
4. Document any new environment variables
5. Use appropriate pytest markers
6. Add realistic timeout values
7. Test both success and failure paths
## Related Documentation
- [Integration Tests](../integration/README.md) - Tests for DB API integration only
- [Unit Tests](../unit/) - Fast isolated tests
- [DB API Contract](../../DB_API_CONTRACT.md) - External DB API specification
Reference in New Issue View Git Blame Copy Permalink