sotopia redis optional

.github/workflows/tests_in_docker.yml

@@ -23,13 +23,15 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-    - name: Checkout
-      uses: actions/checkout@v4
-    - name: Docker Compose
-      run: docker compose -f .devcontainer/docker-compose.yml up -d
-    - name: Run tests
-      run: docker compose -f .devcontainer/docker-compose.yml run --rm -u root -v /home/runner/work/sotopia/sotopia:/workspaces/sotopia devcontainer /bin/sh -c "cd /workspaces/sotopia; ls; uv sync --extra test --extra api; uv run pytest --ignore tests/cli --cov=. --cov-report=xml"
-    - name: Upload coverage report to Codecov
-      uses: codecov/[email protected]
-      with:
-        token: ${{ secrets.CODECOV_TOKEN }}
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Docker Compose
+        run: docker compose -f .devcontainer/docker-compose.yml up -d
+      - name: Run tests with local backend (default)
+        run: docker compose -f .devcontainer/docker-compose.yml run --rm -u root -v /home/runner/work/sotopia/sotopia:/workspaces/sotopia devcontainer /bin/sh -c "cd /workspaces/sotopia; ls; uv sync --extra test --extra api; uv run pytest --ignore tests/cli --cov=. --cov-report=xml"
+      - name: Run database tests with Redis backend
+        run: docker compose -f .devcontainer/docker-compose.yml run --rm -u root -v /home/runner/work/sotopia/sotopia:/workspaces/sotopia devcontainer /bin/sh -c "cd /workspaces/sotopia; SOTOPIA_STORAGE_BACKEND=redis uv run pytest tests/database/ --ignore=tests/database/test_local_storage.py -v"
+      - name: Upload coverage report to Codecov
+        uses: codecov/[email protected]
+        with:
+          token: ${{ secrets.CODECOV_TOKEN }}

README.md

@@ -31,7 +31,6 @@
 Sotopia is an open-ended social learning environment that allows agents to interact with each other and the environment. The environment is designed to be a platform for evaluating and faciliating social intelligence in language agents. The environment is designed to be open-ended, meaning that the environment can be easily extended to include new environments and new agents. The environment is also designed to be scalable, meaning that the environment can be easily scaled to include a large number of agents and environments.
 
 
-
 ```bibtex
 @inproceedings{zhou2024sotopia,
   title = {SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents},
@@ -42,11 +41,7 @@ Sotopia is an open-ended social learning environment that allows agents to inter
 }
 ```
 
-## Help
-See [documentation](https://docs.sotopia.world) for more details.
 
-> [!IMPORTANT]
-> If you are trying to develop on top of Sotopia, we highly recommend to follow the [development guide](https://docs.sotopia.world/contribution/contribution).
 
 ## Get started
 
@@ -60,16 +55,68 @@ We recommend using a virtual environment, e.g. with uv: `pip install uv; uv sync
 Then:
 `uv run sotopia install`
 
+### Storage Backend Options
+
+Sotopia supports two storage backends:
+
+1. **Redis (default)** - Recommended for production use
+   - Requires Redis server running
+   - We recommend using Docker: `docker run -d -p 6379:6379 redis/redis-stack-server:latest`
+   - Set via: `export SOTOPIA_STORAGE_BACKEND=redis` (or leave unset)
+
+2. **Local JSON** - Simpler setup for development/testing
+   - No external dependencies required
+   - Stores data in `~/.sotopia/data/`
+   - Set via: `export SOTOPIA_STORAGE_BACKEND=local`
+   - **Note**: Experimental framework features require Redis
+
 > [!WARNING]
-> We recommend you using docker for setting up the redis server. Other installation methods have been shown to be error-prone.
+> For Redis setup, we recommend using Docker. Other installation methods have been shown to be error-prone.
+
+### Environment Variables
+
+Sotopia uses environment variables for configuration. The recommended way to set them is using a `.env` file in the project root:
+
+```bash
+# Create a .env file
+cat > .env << EOF
+# Required: OpenAI API key
+OPENAI_API_KEY=your_openai_key_here
+
+# Storage backend: "redis" (default) or "local"
+SOTOPIA_STORAGE_BACKEND=local
+
+# Redis connection (only needed if using Redis backend)
+# REDIS_OM_URL=redis://localhost:6379
+EOF
+```
+
+**Environment Variables:**
+- `OPENAI_API_KEY` (required): Your OpenAI API key for running LLM-based simulations
+- `SOTOPIA_STORAGE_BACKEND` (optional): Storage backend - `"redis"` (default) or `"local"`
+- `REDIS_OM_URL` (optional): Redis connection string (default: `"redis://localhost:6379"`)
 
-This will setup the necessary environment variables and download the necessary data.
+**Loading Environment Variables:**
 
-OpenAI key is required to run the code. Please set the environment variable `OPENAI_API_KEY` to your key. The recommend way is to add the key to the conda environment:
+With `uv`:
 ```bash
-conda env config vars set OPENAI_API_KEY=your_key
+# Option 1: Use --env-file flag
+uv run --env-file .env python examples/minimalist_demo.py
+
+# Option 2: Export manually
+export $(cat .env | xargs) && uv run python examples/minimalist_demo.py
+```
+
+With other tools (pip, conda):
+```bash
+# Export variables before running
+export $(cat .env | xargs)
+python examples/minimalist_demo.py
 ```
 
+> [!NOTE]
+> `uv` does not automatically load `.env` files. Use the `--env-file` flag or export variables manually.
+
 ## Easy Sample Server
 You can view an episode demo with default parameters with the following:
 ```python
@@ -95,3 +142,8 @@ or run
 ```bash
 python examples/minimalist_demo.py
 ```
+
+## Help
+
+> [!IMPORTANT]
+> If you are trying to develop on top of Sotopia, we highly recommend to follow the [development guide](https://docs.sotopia.world/contribution/contribution), but cross-reference with this README for latest changes as the documentation may be outdated.

examples/benchmark_evaluator.py

@@ -1,14 +1,14 @@
+import numpy as np
+import pandas as pd
+import scipy.stats as stats
 from evaluate_existing_episode import run_async_server_in_batch_aevaluate
+from typer import Typer
+
 from sotopia.database import (
-    map_human_annotations_to_episode_logs,
     AnnotationForEpisode,
     EpisodeLog,
+    map_human_annotations_to_episode_logs,
 )
-from typer import Typer
-import numpy as np
-import pandas as pd
-import scipy.stats as stats
-
 from sotopia.database.serialization import get_rewards_from_episode
 
 app = Typer()
@@ -158,8 +158,8 @@ def evaluate_evaluator(
             for valid, episode in zip(valid_episodes, re_evaluated_episodes):
                 if not valid:
                     pk = episode.pk  # type: ignore
-                    assert isinstance(pk, str)
-                    to_re_evaluate_list.append(pk)
+                    if pk is not None:
+                        to_re_evaluate_list.append(pk)
 
     correlation_list = []
     ordered_re_eval_episodes = []

pyproject.toml

@@ -55,7 +55,7 @@ dev-dependencies = [
     "lxml-stubs",
     "pandas-stubs",
     "ruff",
-    "mypy"
+    "mypy",
 ]
 
 [tool.mypy]

scripts/README.md

@@ -0,0 +1,165 @@
+# Redis to Local Storage Migration Scripts
+
+This directory contains scripts to migrate data from a Redis dump file to the local JSON storage format.
+
+## Scripts
+
+### `migrate_redis_to_local.sh` (Main Script)
+The master script that orchestrates the entire migration process.
+
+**Usage:**
+```bash
+./scripts/migrate_redis_to_local.sh
+```
+
+**What it does:**
+1. Starts Redis with your dump file (`/Users/xuhuizhou/Downloads/dump_1.rdb`)
+2. Exports all data to `~/.sotopia/data/` in local JSON format
+3. Optionally stops Redis when done
+
+### `start_redis_with_dump.sh`
+Starts a Redis server with the dump file loaded.
+
+**Usage:**
+```bash
+./scripts/start_redis_with_dump.sh
+```
+
+**Features:**
+- Automatically detects and stops existing Redis instances on port 6379
+- Uses `redis-stack-server` if available (required for dump files with Redis modules)
+- Falls back to `redis-server` if redis-stack is not found
+- Creates temporary directory for Redis data
+- Waits for Redis to be ready before exiting
+
+### `export_redis_to_local.py`
+Python script to export all models from Redis to local JSON storage.
+
+**Usage:**
+```bash
+# Basic usage (connects to redis://localhost:6379)
+uv run python scripts/export_redis_to_local.py
+
+# With custom Redis URL
+uv run python scripts/export_redis_to_local.py --redis-url redis://localhost:6380
+
+# With custom output directory
+uv run python scripts/export_redis_to_local.py --output-dir /path/to/output
+
+# Quiet mode (no progress bars)
+uv run python scripts/export_redis_to_local.py --quiet
+```
+
+**Exported Models:**
+- AgentProfile
+- EnvironmentProfile
+- RelationshipProfile
+- EnvAgentComboStorage
+- EnvironmentList
+- Annotator
+- EpisodeLog
+- AnnotationForEpisode
+- NonStreamingSimulationStatus
+
+### `stop_redis.sh`
+Stops the Redis server started by `start_redis_with_dump.sh`.
+
+**Usage:**
+```bash
+./scripts/stop_redis.sh
+```
+
+## Output Format
+
+Data is exported to `~/.sotopia/data/` with the following structure:
+
+```
+~/.sotopia/data/
+├── AgentProfile/
+│   ├── {uuid1}.json
+│   ├── {uuid2}.json
+│   └── ...
+├── EnvironmentProfile/
+│   ├── {uuid1}.json
+│   └── ...
+├── EnvAgentComboStorage/
+│   ├── {uuid1}.json
+│   └── ...
+├── Annotator/
+│   ├── {uuid1}.json
+│   └── ...
+└── ...
+```
+
+Each JSON file contains a single model instance with 2-space indentation:
+
+```json
+{
+  "pk": "01H7VJPFPQ67TTMWZ9246SQ2A4",
+  "env_id": "01H7VFHPJKR16MD1KC71V4ZRCF",
+  "agent_ids": [
+    "01H5TNE5PMBJ9VHH51YC0BB64C",
+    "01H5TNE5P6KZKR2AEY6SZB83H0"
+  ]
+}
+```
+
+## Using the Exported Data
+
+After running the migration, you can use the local storage backend by setting the environment variable:
+
+```bash
+export SOTOPIA_STORAGE_BACKEND=local
+```
+
+Then all Sotopia database operations will use the local JSON files instead of Redis:
+
+```python
+from sotopia.database import AgentProfile, Annotator
+
+# Automatically uses local storage when SOTOPIA_STORAGE_BACKEND=local
+annotators = Annotator.all()
+for annotator in annotators:
+    print(f"{annotator.name}: {annotator.email}")
+```
+
+## Requirements
+
+- Redis 8.0+ or redis-stack-server (for loading dump files with Redis modules)
+- Python 3.10+
+- uv package manager
+- Sotopia dependencies installed (`uv sync --all-extras`)
+
+## Troubleshooting
+
+### "Can't handle RDB format version X"
+Your Redis version is too old. Upgrade to Redis 8.0+:
+```bash
+brew upgrade redis
+```
+
+### "The RDB file contains AUX module data I can't load"
+The dump file was created with redis-stack-server (includes RediSearch, RedisJSON, etc.). Install redis-stack:
+```bash
+brew install redis-stack
+```
+
+### "Redis URL must specify one of the following schemes"
+Make sure the `REDIS_OM_URL` environment variable is set correctly:
+```bash
+export REDIS_OM_URL="redis://localhost:6379"
+```
+
+## Migration Results
+
+After successful migration, you should see output like:
+
+```
+Export Summary
+============================================================
+  ✓ Annotator: 3 records exported (0 errors)
+  ✓ EnvAgentComboStorage: 450 records exported (0 errors)
+  ✓ AnnotationForEpisode: 439 records exported (0 errors)
+============================================================
+Total: 892 records exported (0 errors)
+```