feat(cli): add 'add-all' command and auto_delete_added_files config

[sebastianbraun25]

Problem

The openkb add command supports adding directories recursively, but users who frequently download documents to /raw must manually invoke openkb add raw/ and then manually delete processed files. There is no batch processing command for the /raw staging directory, and no configuration option to auto-delete successfully ingested files and duplicates.

Solution / Changes

New CLI command:

• openkb add-all processes all files in the KB's raw/ directory recursively
• Ingests all supported file types (PDF, Markdown, DOCX, PPTX, XLSX, XLS, HTML, TXT, CSV)
• Returns summary of operation: Added, Skipped, Failed, Deleted counts

New configuration parameter:

• auto_delete_added_files (boolean, default: false)
• When enabled, both add and add-all automatically delete files after processing
• Deletes on both "added" (successful ingestion) and "skipped" (duplicate/already in KB)
• Preserves "failed" files to allow user retries
• Applies to all ingest methods: direct files, directories, and URLs

Implementation:

• Added _delete_if_auto_cleanup_enabled() helper function to handle cleanup logic
• Updated both add and add-all commands to use the helper function
• Updated docstrings to document cleanup behavior on both success and skip cases

Issues

• Resolves feat(cli): add-all command and auto_delete_added_files config #231

[sebastianbraun25]

Rewrote this branch's history to remove unrelated commits.

The branch had accidentally picked up commits from the (now-withdrawn, see
#229/#230) retry-timeout work as ancestors — likely from being built on top
of a local integration branch at the time rather than directly off
main. That made this PR's diff show unrelated changes to
openkb/agent/compiler.py and a new tests/test_compiler_retry.py
that have nothing to do with add-all/auto-delete.

Rebuilt cleanly from current main with only the two commits that
actually belong to this feature (add-all command + the
auto-delete-on-skipped follow-up fix). Diff now only touches
openkb/cli.py and openkb/config.py as expected. No functional
changes to the add-all feature itself — force-pushed the same branch name so
this PR stays open with its history/comments intact.