add simple configuration generator

[sumukshashidhar]

configuration generator for the cli

[alozowski]

This is a good CLI enhancement, but there are a few important points I commented on, especially around API key handling. We should also update tests to cover .env edge cases (.env file is missing or not readable or writable)

[alozowski]

There is a silent failure, and the debugging will be hard. I suggest to log it as always, something like this:

for idx in indices.split(","):
    try:
        i = int(idx.strip()) - 1
        if 0 <= i < len(models):
            selected.append(models[i]["model_name"])
        else:
            logger.warning(f"Model index {idx} is out of range (1-{len(models)})")
    except ValueError:
        logger.warning(f"Invalid model index '{idx}' - expected a number")

[sumukshashidhar]

Fixed! Added proper logging for both out-of-range indices and invalid format errors. Now logs warnings with clear messages about what went wrong.

[alozowski]

Writing here can fail because of the various reasons. Let's wrap into a try/catch

try:
    # the file operations here
except (OSError, PermissionError) as e:
    # log the error using logger.error()
    # exit gracefully with typer.Exit(1)

Also useful if users try to write to read-only folders or forget permissions

[sumukshashidhar]

Wrapped file operations in try/catch block with OSError and PermissionError handling. Shows user-friendly error messages and exits gracefully.

[alozowski]

Right now, if a user enters a real API key instead of a variable like $OPENAI_API_KEY (and there's no input validation) it gets written to config.yaml in plain text. This is a security risk, especially if the file is committed to git (we never know)

I suggest writing to an .env file instead and adding input validation, for example, blocking input that doesn't start with $

[sumukshashidhar]

Added validation to prevent plain text API keys. Now enforces $VAR format and automatically creates/updates .env file with placeholders. Shows clear error if users try to enter real keys.

[alozowski]

Magic numbers make code hard to maintain.

I suggest to place such variables at the top of the file:

DEFAULT_CONCURRENT_REQUESTS_HF = 16
DEFAULT_CONCURRENT_REQUESTS_API = 8
DEFAULT_CHUNK_TOKENS = 256
DEFAULT_MAX_TOKENS = 16384

[sumukshashidhar]

Moved all magic numbers to named constants at the top of the file for better maintainability.

[alozowski]

I noticed we're assigning intfloat/multilingual-e5-large-instruct to the chunking stage by default, but the chunking stage is purely token-based since semantic_chunking was deprecated

Also, this model requires torch and transformers, which aren’t installed in the project, so this could break things or confuse users. Let’s remove it from model_roles["chunking"]?

TODO: update the example/configs/advanced_example.yaml as well and remove semantic_chunking

[sumukshashidhar]

Removed the embedding model from chunking stage since it's purely token-based now. Also cleaned up model_roles to exclude chunking. Will update advanced_example.yaml in a follow-up commit.

[alozowski]

Cool!