CLI Reference
Complete reference guide for the Obvyr CLI tool. For a quick start guide, see Getting Started. For configuration details, see CLI Configuration.
Overview
The Obvyr CLI (obvyr-cli) is a Python command-line tool distributed via PyPI that wraps your test commands to capture execution data and parse JUnit XML output for test-level insights. It works by intercepting command execution, collecting metadata, parsing JUnit XML test results, and sending observations to the Obvyr API for pattern analysis.
How it works:
- Wrap your test command -
obvyr-cli pytest --junitxml=junit.xml tests/ - Generate JUnit XML - Your test framework outputs structured test results
- Obvyr parses JUnit XML - Extracts individual test results, pass/fail status, execution times
- Pattern analysis - Obvyr identifies flaky tests, trends, and reliability metrics
Key characteristics:
- JUnit XML parsing - Core feature that enables test-level insights
- Zero workflow disruption - Tests run exactly as before, even if the profile or API is unavailable
- Language agnostic - Works with any test framework that outputs JUnit XML
- Multi-profile support - Different configurations for different environments
- Attachment governance - Automatic file selection with size limits
Installation
Installation Methods Comparison
| Method | Best For | Pros | Cons | Prerequisites |
|---|---|---|---|---|
| uvx (Recommended) | CI/CD, trying Obvyr, always-latest | No persistent installation, always latest version, fast, no dependency conflicts, downloads Python 3.14 if you do not have it | Slightly longer command syntax | uv installed |
| pipx | Local development, persistent install | Isolated environment, no project conflicts, installed once | Requires manual updates | pipx installed, Python 3.14 or later |
| pip | Simple setups, existing pip workflows | No extra tools, familiar | May conflict with project dependencies | Python 3.14 or later, with pip |
Installing Prerequisites
macOS
bash
# Install uv (for uvx)
brew install uv
# OR install pipx (for pipx)
brew install pipxLinux
bash
# Install uv (for uvx)
curl -LsSf https://astral.sh/uv/install.sh | sh
# OR install pipx (for pipx)
python3 -m pip install --user pipx
python3 -m pipx ensurepathWindows
powershell
# Install uv (for uvx)
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
# OR install pipx (for pipx)
python -m pip install --user pipx
python -m pipx ensurepathInstalling obvyr-cli
Option A: uvx (Recommended)
Run without persistent installation:
bash
# Verify it works
uvx obvyr-cli --help
# Use directly
uvx obvyr-cli pytest tests/Advantages:
- No installation step in CI/CD pipelines
- Always runs latest version automatically
- No PATH configuration needed
- Zero dependency conflicts
- Downloads a suitable Python (3.14 or later) automatically if you do not have one
When to use:
- CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins)
- Trying Obvyr for the first time
- When you want automatic updates
Option B: pipx
Install in isolated environment:
bash
# Install
pipx install obvyr-cli
# Verify
obvyr-cli --help
# Use
obvyr-cli pytest tests/Requires Python 3.14 or later on your PATH.
Advantages:
- Installed once, run many times
- Isolated from project dependencies
- Standard command syntax
When to use:
- Local development with frequent usage
- You prefer persistent installations
- You want explicit version control
Upgrading:
bash
pipx upgrade obvyr-cliUninstalling:
bash
pipx uninstall obvyr-cliOption C: pip
Install in current Python environment:
bash
# Install
pip install obvyr-cli
# Verify
obvyr-cli --help
# Use
obvyr-cli pytest tests/Requires Python 3.14 or later.
Advantages:
- No extra tools required
- Familiar installation method
Disadvantages:
- May conflict with project dependencies
- Installs in current environment
When to use:
- Simple projects without complex dependencies
- Existing pip-based workflows
Upgrading:
bash
pip install --upgrade obvyr-cliUninstalling:
bash
pip uninstall obvyr-cliVerifying Installation
bash
# Check the CLI runs (uvx)
uvx obvyr-cli --help
# Check the CLI runs (pipx/pip)
obvyr-cli --helpEither command prints the usage text and the list of options. To see which version is installed, use pipx list or pip show obvyr-cli.
Platform-Specific Notes
macOS
- Install uv or pipx via Homebrew for easiest setup
- If you use pipx or pip, ensure Python 3.14 or later is installed (
python3 --version). uvx downloads it for you - May need to add pipx binaries to PATH:
export PATH="$PATH:$HOME/.local/bin"
Linux
- Python 3.14 or later required for pipx and pip (uvx downloads it for you)
- On Debian/Ubuntu, your distribution may not package Python 3.14 yet. Use uvx, or install it with
uv python install 3.14 - pipx may require
python3-venvpackage - Add to PATH in
~/.bashrcor~/.zshrc
Windows
- Python 3.14 or later required for pipx and pip (uvx downloads it for you)
- Install from python.org or Microsoft Store
- pipx requires running
pipx ensurepathand restarting terminal - May need to enable script execution:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
Command-Line Interface
Basic Usage
bash
# General syntax
obvyr-cli [OPTIONS] COMMAND [ARGS...]
# With uvx
uvx obvyr-cli [OPTIONS] COMMAND [ARGS...]Global Options
| Option | Short | Description | Example |
|---|---|---|---|
--profile PROFILE | -p | Use named configuration profile | obvyr-cli -p STAGING pytest |
--show-config | - | Display active configuration (API keys hidden) | obvyr-cli --show-config |
--list-profiles | - | List all configured profiles | obvyr-cli --list-profiles |
--verbose | -v | Enable verbose logging (debug mode) | obvyr-cli -v pytest tests/ |
--quiet | -q | Enable quiet mode (errors only) | obvyr-cli -q pytest tests/ |
--dry-run | - | Run the command, then print exactly what would be sent instead of sending it | obvyr-cli --dry-run pytest tests/ |
--output-mode MODE | - | Output mode: auto (detects CI, the default), stream (force streaming), or batch (force batch) | obvyr-cli --output-mode batch pytest |
--no-stream | - | Disable streaming output (equivalent to --output-mode batch) | obvyr-cli --no-stream pytest |
--force-stream | - | Force streaming output (equivalent to --output-mode stream) | obvyr-cli --force-stream pytest |
--no-color | - | Do not set FORCE_COLOR for the wrapped command | obvyr-cli --no-color pytest |
--help | - | Show help message and exit | obvyr-cli --help |
--no-stream and --force-stream cannot be used together. --show-config and --list-profiles only take effect when no command is given.
Put Obvyr's options before the command. Everything from the command onwards is passed to it unchanged, so obvyr-cli -v pytest -p no:cacheprovider turns on Obvyr's verbose logging and gives pytest the -p no:cacheprovider option.
Examples
Basic Usage
bash
# Wrap pytest
obvyr-cli pytest tests/
# Wrap npm test
obvyr-cli npm test
# Wrap gradle test type checking
obvyr-cli gradle test src/
# Wrap any command
obvyr-cli make testUsing Profiles
bash
# Use staging profile
obvyr-cli --profile STAGING pytest tests/
# Short form
obvyr-cli -p PRODUCTION npm test
# Show which profile is active
obvyr-cli --show-config --profile STAGINGVerbose Output
bash
# See detailed execution info
obvyr-cli --verbose pytest tests/
# Combine with profile
obvyr-cli -v -p DEV pytest tests/List Available Profiles
bash
# See all configured profiles
obvyr-cli --list-profiles
# Output example:
# Available profiles:
# - DEFAULT
# - STAGING
# - PRODUCTIONShow Configuration
bash
# Show active configuration (DEFAULT profile)
obvyr-cli --show-config
# Show specific profile configuration
obvyr-cli --show-config --profile PRODUCTION
# Output example (the API key is never printed):
#
# Profile 'PRODUCTION' configuration:
#
# API_URL: https://api.obvyr.com
# ATTACHMENT_PATH: None
# ATTACHMENT_PATHS: []
# ATTACHMENT_MAX_AGE_SECONDS: 10
# TIMEOUT: 10.0
# VERIFY_SSL: True
# ALLOW_INSECURE_API_URL: False
# TAGS: ['production', 'ci']
# ENV_MODE: allowlist
# ENV_INCLUDE: []
# REDACT_PATTERNS: []Exit Codes
The CLI preserves the exit code of the wrapped command:
| Exit Code | Meaning |
|---|---|
0 | Command succeeded |
1-255 | Command failed (exit code from wrapped command) |
1 | Configuration error (if command wasn't run) |
Example:
bash
# If pytest fails with exit code 1, obvyr-cli also exits with 1
obvyr-cli pytest tests/
echo $? # Will show 1 if tests failedThis ensures CI/CD pipelines fail correctly when tests fail.
How It Works
Command Wrapping Architecture
┌─────────────────────────────────────────────┐
│ You run: obvyr-cli pytest tests/ │
└─────────────────┬───────────────────────────┘
│
▼
┌─────────────────────────────────────────────┐
│ 1. CLI loads configuration │
│ - Reads OBVYR_* environment variables │
│ - Validates API key and settings │
└─────────────────┬───────────────────────────┘
│
▼
┌─────────────────────────────────────────────┐
│ 2. CLI executes wrapped command │
│ - Runs pytest tests/ as subprocess │
│ - Captures stdout/stderr in real-time │
│ - Streams output to your terminal │
│ - Records execution timing │
└─────────────────┬───────────────────────────┘
│
▼
┌─────────────────────────────────────────────┐
│ 3. CLI processes output and metadata │
│ - Parses JUnit XML if present │
│ - Selects allow-listed env variables │
│ - Gathers attachment files │
│ - Records exit code and timing │
└─────────────────┬───────────────────────────┘
│
▼
┌─────────────────────────────────────────────┐
│ 4. CLI sends observation to Obvyr API │
│ - Packages all data as JSON │
│ - Authenticates with API key │
│ - Sends asynchronously │
│ - Returns wrapped command exit code │
└─────────────────────────────────────────────┘Data Capture Process
What gets captured:
Command Execution
- Full command string, with secrets in arguments redacted
- Exit code (success/failure)
- Start and end timestamps
- Total execution duration
Output Streams
- Complete stdout and stderr (interleaved), with secrets redacted before upload
- Real-time streaming to terminal (no buffering delay)
Environment Context
- Execution context identifier (
OBVYR_CLI_USER- e.g., "local-dev", "github-ci") - A short allow-list of CI environment variables (build, run, commit, and branch identifiers)
- Anything you add with
ENV_INCLUDE
- Execution context identifier (
Test Framework Metadata
- JUnit XML parsing (if available)
- Individual test results (pass/fail/skip)
- Test execution times
- Test failure messages
File Attachments (optional)
- Test reports
- Coverage data
- Log files
- Any configured attachment paths
What does NOT get captured:
- Environment variables outside the allow-list (unless you add them with
ENV_INCLUDE) - Variables beginning
OBVYR_, including your API key (never sent) - Your operating system, Python version, and working directory
- Files outside configured attachment paths
- Interactive input/output
- Binary files (only text files)
Controlling What Is Sent
The CLI sends only the data described here. Run with --dry-run at any time to see exactly what would be sent.
Environment variables. By default only these CI identifiers are sent. Every other variable is omitted:
CI, CONTINUOUS_INTEGRATION, GITHUB_ACTIONS, GITHUB_RUN_ID, GITHUB_RUN_NUMBER, GITHUB_WORKFLOW, GITHUB_REPOSITORY, GITHUB_SHA, GITHUB_REF_NAME, GITLAB_CI, CI_PIPELINE_ID, CI_JOB_ID, CI_COMMIT_SHA, CI_COMMIT_REF_NAME, JENKINS_URL, BUILD_NUMBER, BUILDKITE, BUILDKITE_BUILD_NUMBER, CIRCLECI, CIRCLE_BUILD_NUM, CIRCLE_SHA1
Variables beginning with OBVYR_ (including your API key) are never sent, even if you name them in ENV_INCLUDE.
| Setting | Default | Description |
|---|---|---|
ENV_MODE | allowlist | allowlist sends the variables above plus ENV_INCLUDE. none sends no environment at all. |
ENV_INCLUDE | [] | Comma-separated names of additional variables to send. |
REDACT_PATTERNS | [] | Comma-separated regular expressions to redact in addition to the built-in patterns. A pattern cannot contain a comma; write [,] instead. |
bash
export OBVYR_PROFILES__DEFAULT__ENV_INCLUDE="BUILD_FLAVOUR,REGION"
export OBVYR_PROFILES__DEFAULT__REDACT_PATTERNS="ACME-\\d+"
export OBVYR_PROFILES__DEFAULT__ENV_MODE="none" # send no environment at allRedaction. Before upload, matches are replaced with [REDACTED] in environment values, command arguments, and command output. The built-in patterns cover:
- Variables and flags whose name contains
token,key,password,passwd,secret, orcredential(the value is redacted) - Assignments such as
password=valueorsecret: value - AWS access key IDs (
AKIA…,ASIA…) - Bearer tokens
Redaction is best-effort pattern matching, not a guarantee. Use --dry-run to check what your commands produce.
What is not changed:
- Terminal output is displayed unredacted. Only the uploaded copy is redacted.
- The wrapped command still receives your full environment.
- Attachments are uploaded as they are, without redaction. Only attach files that are safe to share.
Dry run. --dry-run runs the command as normal, then prints the exact command.json and output.txt that would be uploaded, and lists the attachments by name and size. It makes no network call and does not need an API key to be valid.
Upgrading. Earlier versions sent the entire environment. If you relied on specific variables appearing in Obvyr, add them to ENV_INCLUDE.
API Communication
Request flow:
bash
obvyr-cli pytest tests/
↓
CLI → Obvyr API (https://api.obvyr.com)
↓
Authentication via API key (JWT token)
↓
Observation stored in DynamoDB
↓
Available in dashboard immediatelyAuthentication:
- API keys are 6-month JWT tokens
- Sent in
Authorization: Bearer <token>header - Token includes project_id and agent_id
- Tokens can be rotated in dashboard
Timeout behaviour:
- Default timeout: 10 seconds
- Configurable via
OBVYR_PROFILES__<PROFILE>__TIMEOUT - If API request times out, CLI still succeeds (command result preserved)
Offline Behaviour
What happens if the API is unavailable?
bash
# Your command ALWAYS runs, even if API is down
obvyr-cli pytest tests/
# Scenarios:
# ✅ API available → Observation sent, command runs
# ✅ API timeout → Warning logged, command runs
# ✅ API error → Warning logged, command runs
# ✅ Network down → Warning logged, command runs
# ✅ Profile not found → Warning logged, nothing sent, command runsThe CLI never blocks your workflow. If the API is unreachable, or the profile it needs is not configured, the CLI logs a warning and continues. Your tests run exactly as they would without Obvyr, and the wrapped command's exit code is passed through.
Advanced Usage
Multi-Profile Workflows
Use different profiles for different environments:
bash
# Local development
export OBVYR_PROFILES__DEV__API_KEY="dev-key"
export OBVYR_PROFILES__DEV__TAGS="development,local"
obvyr-cli --profile DEV pytest tests/
# Staging CI
export OBVYR_PROFILES__STAGING__API_KEY="staging-key"
export OBVYR_PROFILES__STAGING__TAGS="staging,ci"
obvyr-cli --profile STAGING pytest tests/
# Production CI
export OBVYR_PROFILES__PROD__API_KEY="prod-key"
export OBVYR_PROFILES__PROD__TAGS="production,ci"
obvyr-cli --profile PROD pytest tests/Why use multiple profiles?
- Track different test types separately (unit, integration, e2e)
- Separate local vs CI observations
- Different projects or services
- Environment-specific configuration
Integration Patterns
Install uv before using uvx
uvx ships with uv and is an alias for uv tool run. It avoids a persistent install of obvyr-cli (no pip install obvyr-cli, no virtual environment, and no dependency conflicts with your project), but it does not avoid installing uv. Hosted images such as ubuntu-latest and python:3.12 do not include uv, so each example below installs it first.
GitHub Actions
yaml
name: Tests with Obvyr
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install dependencies
run: pip install -r requirements.txt
- name: Set up uv
uses: astral-sh/setup-uv@v6
- name: Run tests with Obvyr
env:
OBVYR_CLI_USER: github-ci
OBVYR_PROFILES__DEFAULT__API_KEY: ${{ secrets.OBVYR_API_KEY }}
OBVYR_PROFILES__DEFAULT__TAGS: ci,github-actions
run: uvx obvyr-cli pytest tests/GitLab CI
yaml
test:
stage: test
image: python:3.12
variables:
OBVYR_CLI_USER: gitlab-ci
OBVYR_PROFILES__DEFAULT__TAGS: ci,gitlab
before_script:
- pip install uv
script:
- pip install -r requirements.txt
- export OBVYR_PROFILES__DEFAULT__API_KEY=$OBVYR_API_KEY
- uvx obvyr-cli pytest tests/Jenkins
groovy
pipeline {
agent any
environment {
OBVYR_CLI_USER = 'jenkins-ci'
OBVYR_PROFILES__DEFAULT__API_KEY = credentials('obvyr-api-key')
OBVYR_PROFILES__DEFAULT__TAGS = 'ci,jenkins'
}
stages {
stage('Test') {
steps {
sh 'pip install -r requirements.txt'
sh 'pip install uv'
sh 'uvx obvyr-cli pytest tests/'
}
}
}
}Pre-commit Hook
bash
#!/bin/bash
# .git/hooks/pre-commit
# Run tests with Obvyr before commit
obvyr-cli --profile DEV pytest tests/ || exit 1Custom Configuration Patterns
Per-Repository Configuration
Create .env in project root:
bash
# .env (add to .gitignore!)
OBVYR_CLI_USER=local-dev
OBVYR_PROFILES__DEFAULT__API_KEY=your-api-key
OBVYR_PROFILES__DEFAULT__TAGS=my-project,localShell Aliases
bash
# ~/.bashrc or ~/.zshrc
# Alias for common test command
alias otest="obvyr-cli pytest tests/"
# Alias for verbose testing
alias otestv="obvyr-cli --verbose pytest tests/"
# Alias for staging profile
alias otest-staging="obvyr-cli --profile STAGING pytest tests/"Makefile Integration
makefile
# Makefile
.PHONY: test test-staging test-prod
test:
obvyr-cli pytest tests/
test-staging:
obvyr-cli --profile STAGING pytest tests/
test-prod:
obvyr-cli --profile PROD pytest tests/
test-verbose:
obvyr-cli --verbose pytest tests/Performance Considerations
CLI overhead:
- Typical overhead: 100-300ms per command execution
- Primarily from API request (async, non-blocking)
- JUnit XML parsing adds ~50ms for large test suites
Minimising overhead:
- Use uvx in CI/CD (no installation time)
- Ensure good network connectivity to API
- Limit attachment file sizes
- Use tags instead of excessive metadata
Attachment performance:
- Files validated before upload (existence, size, freshness)
- Priority-based selection (JSON > XML > YAML > plain text)
- 5MB per file limit, 10MB total limit
- Binary files automatically excluded
Troubleshooting
Installation Issues
"Command not found: obvyr-cli" (pipx/pip)
Cause: Installation directory not in PATH
Solution:
bash
# Check where pipx installs binaries
pipx list
# Add to PATH (add to ~/.bashrc or ~/.zshrc)
export PATH="$PATH:$HOME/.local/bin"
# Reload shell
source ~/.bashrc # or source ~/.zshrc"Command not found: uvx"
Cause: uv not installed
Solution:
bash
# macOS
brew install uv
# Linux
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex""Python version too old"
Cause: obvyr-cli requires Python 3.14 or later. pipx and pip use the Python already on your machine.
Solution: run the CLI with uvx, which downloads a suitable Python for you:
bash
uvx obvyr-cli pytest tests/Or install Python 3.14 or later yourself and keep using pipx or pip:
bash
# Check current version
python3 --version
# Install Python 3.14 (macOS)
brew install python@3.14
# Install Python 3.14 with uv (any platform)
uv python install 3.14Configuration Issues
"Profile 'DEFAULT' not found"
What you see: a warning saying nothing is being reported to Obvyr. Your command still runs and its exit code is passed through, but no observation is sent.
Cause: OBVYR_PROFILES__DEFAULT__API_KEY not set
Solution:
bash
# Check if set
echo $OBVYR_PROFILES__DEFAULT__API_KEY
# Set it
export OBVYR_PROFILES__DEFAULT__API_KEY="your-api-key"
# Or create .env file
cat > .env << EOF
OBVYR_PROFILES__DEFAULT__API_KEY=your-api-key
OBVYR_CLI_USER=local-dev
EOF"Profile 'STAGING' not found"
What you see: the same warning, naming the profile you asked for with --profile. Your command still runs, but nothing is sent. (--show-config has nothing to show for a missing profile, so it reports Error: Profile 'STAGING' not found in configuration. instead.)
Cause: Named profile not configured
Solution:
bash
# List available profiles
obvyr-cli --list-profiles
# Either:
# 1. Configure the missing profile
export OBVYR_PROFILES__STAGING__API_KEY="staging-key"
# 2. Use an existing profile
obvyr-cli --profile DEFAULT pytest tests/
# 3. Use default profile (omit --profile flag)
obvyr-cli pytest tests/"Invalid API key" or 401 Unauthorized
Cause: Incorrect or expired API key
Solution:
- Verify API key in dashboard (Project → Agent → Settings)
- Check for typos in environment variable
- Regenerate API key if needed
- Ensure no extra whitespace in key
bash
# Verify what's configured
obvyr-cli --show-config
# Check environment variable
echo "$OBVYR_PROFILES__DEFAULT__API_KEY" | cat -ARuntime Issues
"Connection timeout" or "Cannot reach API"
Cause: Network connectivity issues
Solution:
bash
# Test connectivity
curl https://api.obvyr.com/health
# Check proxy settings
echo $HTTP_PROXY
echo $HTTPS_PROXY
# If behind proxy, configure
export HTTPS_PROXY=http://proxy.example.com:8080"SSL certificate verification failed"
Cause: Corporate proxy or SSL interception
Solution (temporary):
bash
# Disable SSL verification (NOT recommended for production)
export OBVYR_PROFILES__DEFAULT__VERIFY_SSL=falseSolution (proper):
bash
# Install corporate CA certificate
export REQUESTS_CA_BUNDLE=/path/to/corporate-ca.crtTests fail when wrapped but pass normally
Cause: Rare output buffering or environment variable conflicts
Solution:
bash
# Run with verbose logging to diagnose
obvyr-cli --verbose pytest tests/
# Check environment variables
obvyr-cli env | grep OBVYR
# Report issue with verbose outputDebug Mode
Enable verbose logging for troubleshooting:
bash
# Verbose CLI output
obvyr-cli --verbose pytest tests/
# Set log level via environment variable
export OBVYR_LOG_LEVEL=DEBUG
obvyr-cli pytest tests/
# Combine with profile debugging
obvyr-cli -v --show-config --profile STAGINGWhat verbose mode shows:
- Configuration loading details
- Command execution timing
- API request/response details
- Attachment file selection process
- JUnit XML parsing results
Getting Help
If you encounter issues not covered here:
- Check configuration:
obvyr-cli --show-config - Run with verbose output:
obvyr-cli --verbose pytest tests/ - Check API status: Visit status.obvyr.com (if available)
- Review logs: Look for warning/error messages
- Contact support: support@obvyr.com
Include in your support request:
- CLI version (
pipx listorpip show obvyr-cli; for uvx, say that you use uvx and the date you last ran it) - Python version (
python3 --version) - Operating system
- Installation method (uvx/pipx/pip)
- Verbose output (
--verbose) - Configuration (with API key redacted)
Next Steps
- Getting Started - Quick 10-minute setup guide
- CLI Configuration - Environment variables and advanced configuration
- Why Obvyr? - Understand the value proposition
- Roadmap - See what's coming next