35f31973c4
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: thomasnordquist <7721625+thomasnordquist@users.noreply.github.com> Co-authored-by: Thomas Nordquist <thomasnordquist@users.noreply.github.com>
88 lines
2.6 KiB
Markdown
88 lines
2.6 KiB
Markdown
# LLM Tests Debugging Summary
|
|
|
|
## GitHub Workflow Issue Fixed ✅
|
|
|
|
### Problem Identified
|
|
The `.github/workflows/copilot-setup-steps.yml` had a critical step ordering issue:
|
|
|
|
**Before (BROKEN):**
|
|
1. Create `.env.llm-tests` file
|
|
2. Checkout code ← **This overwrites the directory, losing the .env file!**
|
|
3. Run tests
|
|
|
|
**After (FIXED):**
|
|
1. Checkout code
|
|
2. Create `.env.llm-tests` file ← **Now persists correctly**
|
|
3. Run tests
|
|
|
|
### Changes Made
|
|
- Moved "Persist Secrets to Agent Environment" step AFTER "Checkout code"
|
|
- Added `export` prefix to environment variables for proper shell sourcing
|
|
- Added `RUN_LLM_TESTS=true` to enable tests automatically
|
|
- Added `chmod 600` for security
|
|
- Added verification logging to confirm file creation
|
|
|
|
## Environment Setup Verification
|
|
|
|
### API Key Sourcing ✅
|
|
The `.env.llm-tests` sourcing mechanism works correctly:
|
|
|
|
```bash
|
|
# Create .env file
|
|
echo 'export OPENAI_API_KEY=sk-your-key' > .env.llm-tests
|
|
echo 'export RUN_LLM_TESTS=true' >> .env.llm-tests
|
|
|
|
# Source and verify
|
|
source .env.llm-tests
|
|
echo $OPENAI_API_KEY # Shows the key
|
|
```
|
|
|
|
### Test Detection ✅
|
|
When the environment is properly sourced, tests correctly:
|
|
- Detect the API key presence
|
|
- Enable live test execution (not skipped)
|
|
- Show provider detection: "Running LLM integration tests with provider: openai"
|
|
|
|
### Current Limitation ⚠️
|
|
Tests fail in the jsdom environment with network errors:
|
|
```
|
|
Error: Cross origin null forbidden
|
|
Error: LLM API call failed: Network Error
|
|
```
|
|
|
|
This is expected because:
|
|
1. Tests run in a jsdom environment (not a real browser)
|
|
2. axios HTTP requests fail due to CORS restrictions in jsdom
|
|
3. Live API tests need a proper Node.js environment or network mocking
|
|
|
|
## Recommendations
|
|
|
|
### For Local Development
|
|
Run tests with a real API key in a Node environment:
|
|
```bash
|
|
source .env.llm-tests
|
|
cd app && yarn test
|
|
```
|
|
|
|
### For CI/CD
|
|
The workflow now correctly:
|
|
1. Checks out the repository first
|
|
2. Creates `.env.llm-tests` in the workspace
|
|
3. Makes the API key available to subsequent steps
|
|
|
|
Consider:
|
|
1. Running tests in a Node environment (not jsdom)
|
|
2. Using nock or msw to mock HTTP requests in tests
|
|
3. Running live tests only in scheduled jobs with proper network access
|
|
|
|
## Verified Working
|
|
- ✅ `.env.llm-tests` creation via workflow (step order fixed)
|
|
- ✅ `.env.llm-tests` creation via `setup-llm-env.sh`
|
|
- ✅ Environment variable sourcing
|
|
- ✅ Test detection of API keys
|
|
- ✅ Provider auto-detection (OpenAI/Gemini)
|
|
- ✅ Proper skip behavior when no API key
|
|
|
|
## Status
|
|
The infrastructure is now working correctly. The workflow step order has been fixed to ensure `.env.llm-tests` persists after checkout.
|