# CoAiAPy

> Python package for AI workflow automation and observability with revolutionary pipeline templates and environment management

CoAiAPy is a comprehensive Python package that transforms complex AI workflow creation from 30+ minute manual processes into 30-second automated pipelines. It provides audio transcription, synthesis, Redis data stashing, and complete Langfuse integration for AI pipeline observability.

## Key Features

- **Pipeline Templates**: 5 built-in templates with Jinja2 variable substitution and conditional logic
- **Environment Management**: Persistent cross-session variables with `.coaia-env` files  
- **Langfuse Integration**: Complete CLI for traces, observations, prompts, and datasets
- **Audio Processing**: AWS Polly transcription and synthesis
- **Redis Integration**: Key-value stashing with TTL support
- **Python 3.6 Compatible**: Works with Pythonista iOS app

## Installation

```bash
pip install coaiapy
```

## Core CLI Commands

### Quick Start
```bash
coaia --help                    # Show all commands
coaia init                      # Create sample config file
```

### Pipeline Templates (Revolutionary Feature)
```bash
# List available templates
coaia pipeline list
coaia pipeline list --path --json

# Show template details
coaia pipeline show simple-trace
coaia pipeline show data-pipeline --preview

# Create automated workflows (30+ min → 30 sec)
coaia pipeline create simple-trace --var trace_name="My Process" --var user_id="john"
coaia pipeline create data-pipeline --var pipeline_name="ETL Process" --export-env

# Create custom templates
coaia pipeline init my-template
coaia pipeline init custom-workflow --from data-pipeline --location project
```

### Environment Management
```bash
# Initialize persistent environment
coaia env init                         # Project .coaia-env
coaia env init --global               # User ~/.coaia/global.env
coaia env init --name dev             # Named environment

# Manage variables
coaia env set COAIA_USER_ID "john"    # Persist to file
coaia env get COAIA_TRACE_ID          # Get variable
coaia env list                        # Show all environments

# Shell integration
eval $(coaia env source --export)    # Load into shell
coaia env save --name "my-context"   # Save current state
```

### Audio Processing
```bash
coaia transcribe audio.mp3           # Transcribe audio to text
coaia transcribe audio.mp3 -O output.txt
```

### Text Processing
```bash
coaia summarize "long text here"     # Summarize text
coaia summarize --f input.txt        # Summarize from file
coaia p dictkore "text to correct"   # Custom process tags
```

### Redis Operations
```bash
coaia tash mykey "value"             # Stash key-value
coaia tash mykey --file data.txt     # Stash from file
coaia fetch mykey                    # Retrieve value
coaia fetch mykey --output save.txt # Save to file
```

### Langfuse Integration (AI Observability)

#### Prompts Management
```bash
coaia fuse prompts list              # List all prompts
coaia fuse prompts get MyPrompt      # Get latest prompt
coaia fuse prompts get MyPrompt --prod  # Production version
coaia fuse prompts get MyPrompt -c   # Content only
coaia fuse prompts get MyPrompt -e   # Escaped single line
coaia fuse prompts create MyPrompt "content" --labels prod dev
```

#### Datasets Management
```bash
coaia fuse datasets list             # List all datasets
coaia fuse datasets get MyDataset    # Get dataset with items
coaia fuse datasets get MyDataset --json
coaia fuse datasets get MyDataset -oft > training.jsonl  # OpenAI format
coaia fuse datasets get MyDataset -gft > training.jsonl  # Gemini format
coaia fuse datasets create NewDataset --description "My dataset"
```

#### Traces & Observations (Production AI Pipeline Support)
```bash
# Create trace with environment export
eval $(coaia fuse traces create $(uuidgen) -s $(uuidgen) -u pipeline-user -n "AI Workflow" --export-env)

# Add observations with shortcuts
coaia fuse traces add-observation $COAIA_TRACE_ID -te -n "Data Loaded"    # EVENT
coaia fuse traces add-observation $COAIA_TRACE_ID -ts -n "Processing"     # SPAN  
coaia fuse traces add-observation $COAIA_TRACE_ID -tg -n "LLM Call" --model "gpt-4"  # GENERATION

# Parent-child relationships
eval $(coaia fuse traces add-observation $COAIA_TRACE_ID -ts -n "Main Workflow" --export-env)
coaia fuse traces add-observation $COAIA_TRACE_ID -n "Step 1" --parent $COAIA_LAST_OBSERVATION_ID

# Batch operations
coaia fuse traces add-observations $COAIA_TRACE_ID -f observations.json
cat observations.yaml | coaia fuse traces add-observations $COAIA_TRACE_ID --format yaml
```

#### Score Management
```bash
coaia fuse scores list               # List scores
coaia fuse score-configs list        # List score configurations
coaia fuse score-configs create "Quality" NUMERIC --min-value 0 --max-value 10
coaia fuse score-configs presets list  # Built-in presets
```

## Built-in Pipeline Templates

1. **simple-trace**: Basic monitoring with single observation
2. **data-pipeline**: Multi-step data processing with validation  
3. **llm-chain**: LLM interaction pipeline with input/output tracking
4. **parallel-processing**: Concurrent task execution with synchronization
5. **error-handling**: Robust error management with retry mechanisms

## Template Features

- **Jinja2 Templating**: Variable substitution with `{{ variable_name }}`
- **Built-in Functions**: `uuid4()`, `now()`, `timestamp()` available
- **Conditional Steps**: `{% if condition %}...{% endif %}`
- **Parent-Child Hierarchies**: Automatic SPAN observation relationships
- **Auto-Generation**: Trace IDs, observation IDs, timestamps
- **Template Hierarchy**: Project → Global → Built-in discovery

## Environment Configuration

### Required Environment Variables
```bash
# AWS Services (audio transcription/synthesis)
OPENAI_API_KEY
AWS_KEY_ID
AWS_SECRET_KEY  
AWS_REGION

# Redis (data stashing)
REDIS_HOST
REDIS_PORT
REDIS_PASSWORD
REDIS_SSL

# Langfuse (AI observability)
LANGFUSE_SECRET_KEY
LANGFUSE_PUBLIC_KEY
LANGFUSE_HOST
```

### Configuration Files
- `~/coaia.json` - Main configuration with process tags and API settings
- `.coaia-env` - Project environment variables (JSON or .env format)
- `~/.coaia/global.env` - Global environment variables
- `.coaia/templates/` - Custom pipeline templates

## Advanced Workflows

### One-Command Pipeline Creation
```bash
# Before: Complex 30+ minute manual setup
export TRACE_ID=$(uuidgen)
export SESSION_ID=$(uuidgen)
coaia fuse traces create $TRACE_ID -u john -s $SESSION_ID
export OBS_ID=$(uuidgen)
coaia fuse traces add-observation $OBS_ID $TRACE_ID -ts -n "Step 1"
# ... repeat for each step ...

# After: 30-second automation
coaia pipeline create data-pipeline \
  --var user_id="john" \
  --var pipeline_name="ETL Process" \
  --export-env
```

### Cross-Session Workflow Persistence
```bash
# Session 1: Start and persist
coaia pipeline create llm-chain --var model="gpt-4" --export-env
coaia env save --name "llm-session"

# Session 2: Resume seamlessly (hours/days later)
eval $(coaia env source --name llm-session --export)
coaia fuse traces add-observation $COAIA_TRACE_ID -n "Continued processing"
```

### Custom Template Creation
```bash
# Create project-specific template
coaia pipeline init company-etl --from data-pipeline --location project

# Template saved to ./.coaia/templates/company-etl.json
# Edit with custom variables and steps

# Use custom template
coaia pipeline create company-etl --var data_source="production_db"
```

## Python Integration

```python
import coaiapy
from coaiapy import EnvironmentManager, TemplateLoader

# Environment management
env_manager = EnvironmentManager()
env_vars = env_manager.load_environment()

# Template loading
loader = TemplateLoader()
template = loader.load_template("data-pipeline")
```

## Development & Building

```bash
make clean          # Remove build artifacts
make bump           # Auto-increment version
make build          # Create wheel and source distributions
make test-release   # Full automation: bump + clean + build + upload to TestPyPI
make upload         # Upload to production PyPI
```

## Links

- [GitHub Repository](https://github.com/jgwill/coaiapy)
- [PyPI Package](https://pypi.org/project/coaiapy/)
- [llms.txt (AI Documentation)](https://coaiapy.jgwill.com/llms.txt)
- [Documentation Wiki](https://github.com/jgwill/coaiapy/wiki)
- [Pipeline Templates Guide](https://github.com/jgwill/coaiapy/wiki/Pipeline-Templates)
- [Environment Management Guide](https://github.com/jgwill/coaiapy/wiki/Environment-Management)
- [Langfuse Integration Guide](https://github.com/jgwill/coaiapy/wiki/Langfuse-Integration)

## Python Compatibility

- **Minimum**: Python >=3.6 (Pythonista iOS compatible)
- **Dependencies**: boto3<=1.26.137, mutagen<=1.45.1, redis==5.1.1, jinja2>=2.10
- **Platforms**: Linux, macOS, Windows, iOS (Pythonista)