Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Inspired by [code-review-graph](https://github.com/tirth8205/code-review-graph)
- **Evidence-driven code exploration**: DB-backed retrieval returns small file-level candidates with matched fields, evidence nodes, and optional docs before agents drill into exact graph nodes
- **Browser Wiki UI**: `ccg-server` can serve generated docs, tree search, DB-backed retrieval, Context Tray copying, and an Obsidian-style graph viewer
- **Custom annotations**: `@intent`, `@domainRule`, `@sideEffect`, `@mutates`, `@index` — search code by business context ([details](guide/annotations.md))
- **Webhook sync**: GitHub / Gitea push events → auto clone + build with per-repo branch filtering and `.ccg.yaml` `include_paths` auto-loading ([details](guide/webhook.md))
- **Webhook sync**: GitHub / Gitea push events → auto clone + build with per-repo branch filtering and `.ccg.yaml` build-scope (`include_paths` / `exclude`) auto-loading ([details](guide/webhook.md))
- **Multi-DB**: SQLite (local), PostgreSQL
- **Full-text search**: FTS5 (SQLite), tsvector+GIN (PostgreSQL)

Expand Down
15 changes: 14 additions & 1 deletion guide/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,7 +198,7 @@ docs:
Restricts the build target paths. When set, only paths under the specified directories are parsed.

- CLI: `.ccg.yaml`'s `include_paths` is automatically applied during `ccg build`
- Webhook: After cloning a repo, `.ccg.yaml`'s `include_paths` is auto-loaded to limit build scope
- Webhook: After cloning a repo, `.ccg.yaml`'s `include_paths` and `exclude` are auto-loaded to set build scope
- Incremental build (`ccg update`): `include_paths` filter applied when collecting changed files

```yaml
Expand All @@ -207,6 +207,19 @@ include_paths:
- src/shared/
```

### `exclude`

Filters matching files and directories from the build scope. It supports path prefixes, filename globs, full-path globs, and regular expressions.

- CLI and incremental builds apply `exclude` while collecting source files.
- Webhook builds load `exclude` from the cloned repository's root `.ccg.yaml`.

```yaml
exclude:
- vendor/
- "*_generated.go"
```

### Regex Patterns

The `exclude` and `rules` pattern fields support regular expressions. Patterns containing `$`, `^`, `+`, `{}`, `|`, `\.`, `.*` are automatically detected as regex:
Expand Down
15 changes: 14 additions & 1 deletion guide/ko/cli-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ docs:
빌드 대상 경로를 제한합니다. 설정된 경우 지정된 디렉토리 하위의 경로만 파싱됩니다.

- CLI: `ccg build` 중에 `.ccg.yaml`의 `include_paths`가 자동으로 적용됩니다.
- 웹훅: 저장소를 복제한 후 빌드 범위를 제한하기 위해 `.ccg.yaml`의 `include_paths`를 자동으로 로드합니다.
- 웹훅: 저장소를 복제한 후 빌드 범위를 설정하기 위해 `.ccg.yaml`의 `include_paths`와 `exclude`를 자동으로 로드합니다.
- 증분 빌드(`ccg update`): 변경된 파일을 수집할 때 `include_paths` 필터가 적용됩니다.

```yaml
Expand All @@ -187,6 +187,19 @@ include_paths:
- src/shared/
```

### `exclude`

빌드 범위에서 일치하는 파일과 디렉터리를 제외합니다. 경로 접두사, 파일명 glob, 전체 경로 glob, 정규 표현식을 지원합니다.

- CLI와 증분 빌드는 소스 파일 수집 과정에서 `exclude`를 적용합니다.
- 웹훅 빌드는 복제된 저장소 루트의 `.ccg.yaml`에서 `exclude`를 로드합니다.

```yaml
exclude:
- vendor/
- "*_generated.go"
```

### 정규식 패턴 (Regex Patterns)

`exclude` 및 `rules` 패턴 필드는 정규 표현식을 지원합니다. `$`, `^`, `+`, `{}`, `|`, `\.`, `.*`를 포함하는 패턴은 자동으로 정규식으로 감지됩니다.
Expand Down
17 changes: 11 additions & 6 deletions guide/ko/webhook.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,28 +170,33 @@ CCG를 Streamable HTTP로 실행하면 exporter가 설정되지 않은 경우에
- 지수적 증가: 1s → 2s → 4s → ... (MaxDelay에서 캡핑)
- 대기 중인 재시도는 컨텍스트 취소(서버 종료) 시 즉시 취소됩니다.
- 패닉(Panic)은 오류로 처리되어 재시도 대상이 됩니다.
- 잘못된 `.ccg.yaml` `include_paths` 설정과 같이 유효하지 않은 저장소 설정은 현재 이벤트에 대해 재시도 불가능한 오류로 처리됩니다.
- 잘못된 `.ccg.yaml` `include_paths` 또는 `exclude`처럼 유효하지 않은 저장소 빌드 범위 설정은 현재 이벤트에 대해 재시도 불가능한 오류로 처리됩니다.
- MaxAttempts를 초과하면 `ERROR` 로그를 남기고 동기화를 포기합니다 (다음 push 이벤트 시 재시도 가능).

이러한 기본값은 `--webhook-attempt-timeout`, `--webhook-retry-attempts`, `--webhook-retry-base-delay`, `--webhook-retry-max-delay`로 조정할 수 있습니다.

## `.ccg.yaml` include_paths 자동 적용
## `.ccg.yaml` 빌드 범위 자동 적용

웹훅 빌드 중에 복제된 저장소 내부의 `.ccg.yaml`에 있는 `include_paths` 설정을 자동으로 읽어 빌드 범위를 제한합니다.
웹훅 빌드 중에 복제된 저장소 루트의 `.ccg.yaml`에서 `include_paths`와 `exclude`를 자동으로 읽습니다. `include_paths`는 빌드 범위를 좁히고, `exclude`는 그 범위에서 일치하는 파일 또는 디렉터리를 제외합니다.

```yaml
# 저장소 내부의 .ccg.yaml
include_paths:
- src/
- lib/

exclude:
- vendor/
- "*_generated.go"
```

- `.ccg.yaml`이 없거나 `include_paths` 키가 없는 경우 전체 디렉토리가 빌드됩니다.
- `.ccg.yaml`이 없으면 추가 제외 없이 전체 디렉토리가 빌드됩니다.
- 두 키는 각각 선택 사항입니다. `include_paths`가 없으면 전체 저장소가 대상이고, `exclude`가 없으면 추가 경로 필터링이 없습니다.
- CLI의 `--config` 플래그와 무관하게 작동합니다 (YAML 직접 파싱).

## 파싱 크기 제한 (Parse Size Limits)

웹훅 요청 본문의 크기는 저장소 파싱과는 별도로 제한됩니다. 웹훅 페이로드는 서버에 의해 제한되지만, 이후의 복제/빌드 단계는 기본적으로 소스 파싱 크기 제한이 없습니다. 기본적으로 CCG는 `include_paths`로 범위를 좁히지 않는 한 복제된 저장소의 모든 일치하는 소스 파일을 빌드합니다.
웹훅 요청 본문의 크기는 저장소 파싱과는 별도로 제한됩니다. 웹훅 페이로드는 서버에 의해 제한되지만, 이후의 복제/빌드 단계는 기본적으로 소스 파싱 크기 제한이 없습니다. 기본적으로 CCG는 `include_paths`로 범위를 좁히거나 `exclude`로 경로를 제외하지 않는 한 복제된 저장소의 모든 일치하는 소스 파일을 빌드합니다.

대규모 저장소에 대해 파싱 예산이 필요한 경우, `--max-file-bytes`, `--max-total-parsed-bytes` 또는 일치하는 `.ccg.yaml` 설정을 명시적으로 구성하십시오. CCG는 기본 웹훅 파싱 제한을 두지 않습니다.

Expand Down Expand Up @@ -225,7 +230,7 @@ CCG는 현재 웹훅 운영을 위한 `/metrics` 엔드포인트를 제공하지
`/status`가 degraded 웹훅 동기화를 보고하거나, 큐 대기 시간이 계속 증가하거나, 배포 재시작 중 수락된 이벤트가 중단되었을 수 있을 때는 아래 절차를 사용하십시오.

1. `/status.webhook.recent_repos`와 로그에서 `repo`, `branch`, `last_error`를 확인합니다.
2. 잘못된 `.ccg.yaml` `include_paths`처럼 재시도 불가능한 저장소 설정 오류를 수정합니다.
2. 잘못된 `.ccg.yaml` `include_paths` 또는 `exclude`처럼 재시도 불가능한 저장소 빌드 범위 설정 오류를 수정합니다.
3. upstream provider가 복구를 주도해야 한다면 같은 브랜치에 새 push를 발생시킵니다.
4. 수동 복구가 필요하면 checkout 디렉터리 기준으로 네임스페이스를 업데이트합니다.

Expand Down
16 changes: 11 additions & 5 deletions guide/webhook.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,33 +190,39 @@ On clone or build failure, automatically retries with exponential backoff:
- Exponential growth: 1s → 2s → 4s → ... (capped at MaxDelay)
- Pending retries are immediately cancelled on context cancellation (server shutdown)
- Panics are treated as errors and are eligible for retry
- Invalid repository config, such as malformed `.ccg.yaml` `include_paths`, is treated as non-retryable for the current event
- Invalid repository build-scope config, such as malformed `.ccg.yaml` `include_paths` or `exclude`, is treated as non-retryable for the current event
- After exceeding MaxAttempts, logs an `ERROR` and abandons the sync (retryable on next push event)

These defaults can be tuned with `--webhook-attempt-timeout`,
`--webhook-retry-attempts`, `--webhook-retry-base-delay`, and
`--webhook-retry-max-delay`.

## `.ccg.yaml` include_paths Auto-Apply
## `.ccg.yaml` Build Scope Auto-Apply

During webhook builds, the `include_paths` setting from `.ccg.yaml` inside the cloned repo is automatically read to restrict build scope.
During webhook builds, CCG reads `include_paths` and `exclude` from `.ccg.yaml` at the cloned repository root. `include_paths` narrows the build scope, and `exclude` removes matching files or directories from that scope.

```yaml
# .ccg.yaml inside the repo
include_paths:
- src/
- lib/

exclude:
- vendor/
- "*_generated.go"
```

- If `.ccg.yaml` is absent or has no `include_paths` key, the entire directory is built
- If `.ccg.yaml` is absent, CCG builds the entire directory without additional exclusions
- Each key is optional: no `include_paths` means the whole repository is eligible, while no `exclude` means no extra path filtering
- Operates independently of the CLI's `--config` flag (direct YAML parsing, no viper)

## Parse Size Limits

Webhook request bodies are limited separately from repository parsing. The
webhook payload is capped by the server, but the subsequent clone/build step has
no default source parse size limit. By default, CCG builds every matching source
file in the cloned repository unless `include_paths` narrows the scope.
file in the cloned repository unless `include_paths` narrows the scope or
`exclude` filters paths out.

If a deployment needs a parse budget for large repositories, configure it
explicitly with `--max-file-bytes`, `--max-total-parsed-bytes`, or the matching
Expand Down
31 changes: 16 additions & 15 deletions internal/adapters/outbound/configfiles/includes.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,32 +7,33 @@ import (
"path/filepath"

"go.yaml.in/yaml/v3"

"github.com/tae2089/code-context-graph/internal/app/reposync"
)

var errIncludePathsConfig = errors.New("invalid include_paths config")
var errBuildScopeConfig = errors.New("invalid repository build scope config")

// IncludePaths reads the repository-local CCG include path configuration.
// @intent adapt established repository config parsing to the reposync application port.
type IncludePaths struct{}
// BuildScope reads repository-local CCG source selection configuration.
// @intent adapt repository include and exclude configuration parsing to the reposync application port.
type BuildScope struct{}

// Load reads include paths from a repository-local .ccg.yaml file.
// @intent own repository configuration file I/O for the reposync adapter while preserving absent-file and parse-error behavior.
func (IncludePaths) Load(repoDir string) ([]string, error) {
// Load reads include paths and exclude patterns from a repository-local .ccg.yaml file.
// @intent own repository build scope configuration I/O for webhook synchronization.
// @return returns an empty scope when the repository has no .ccg.yaml file.
func (BuildScope) Load(repoDir string) (reposync.BuildScope, error) {
data, err := os.ReadFile(filepath.Join(repoDir, ".ccg.yaml"))
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return nil, nil
return reposync.BuildScope{}, nil
}
return nil, err
return reposync.BuildScope{}, err
}
var cfg struct {
IncludePaths []string `yaml:"include_paths"`
IncludePaths []string `yaml:"include_paths"`
ExcludePatterns []string `yaml:"exclude"`
}
if err := yaml.Unmarshal(data, &cfg); err != nil {
return nil, fmt.Errorf("%w: parse .ccg.yaml: %w", errIncludePathsConfig, err)
}
if len(cfg.IncludePaths) == 0 {
return nil, nil
return reposync.BuildScope{}, fmt.Errorf("%w: parse .ccg.yaml: %w", errBuildScopeConfig, err)
}
return cfg.IncludePaths, nil
return reposync.BuildScope{IncludePaths: cfg.IncludePaths, ExcludePatterns: cfg.ExcludePatterns}, nil
}
43 changes: 37 additions & 6 deletions internal/adapters/outbound/configfiles/includes_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
"github.com/tae2089/code-context-graph/internal/adapters/outbound/configfiles"
)

func TestIncludePathsLoad(t *testing.T) {
func TestBuildScopeLoad_IncludePaths(t *testing.T) {
tests := []struct {
name string
content *string
Expand All @@ -30,20 +30,51 @@ func TestIncludePathsLoad(t *testing.T) {
}
}

got, err := (configfiles.IncludePaths{}).Load(dir)
got, err := (configfiles.BuildScope{}).Load(dir)
if (err != nil) != tt.wantErr {
t.Fatalf("Load() error = %v, wantErr %v", err, tt.wantErr)
}
if len(got) != len(tt.want) {
t.Fatalf("Load() = %v, want %v", got, tt.want)
if len(got.IncludePaths) != len(tt.want) {
t.Fatalf("IncludePaths = %v, want %v", got.IncludePaths, tt.want)
}
for i := range tt.want {
if got[i] != tt.want[i] {
t.Fatalf("Load()[%d] = %q, want %q", i, got[i], tt.want[i])
if got.IncludePaths[i] != tt.want[i] {
t.Fatalf("IncludePaths[%d] = %q, want %q", i, got.IncludePaths[i], tt.want[i])
}
}
})
}
}

func TestBuildScopeLoad_ProvidesExcludePatterns(t *testing.T) {
dir := t.TempDir()
content := "include_paths:\n - cmd\n - internal\nexclude:\n - vendor\n - \"*_generated.go\"\n"
if err := os.WriteFile(filepath.Join(dir, ".ccg.yaml"), []byte(content), 0o644); err != nil {
t.Fatal(err)
}

scope, err := (configfiles.BuildScope{}).Load(dir)
if err != nil {
t.Fatalf("Load() error = %v", err)
}
wantIncludes := []string{"cmd", "internal"}
if len(scope.IncludePaths) != len(wantIncludes) {
t.Fatalf("IncludePaths = %v, want %v", scope.IncludePaths, wantIncludes)
}
for i := range wantIncludes {
if scope.IncludePaths[i] != wantIncludes[i] {
t.Fatalf("IncludePaths[%d] = %q, want %q", i, scope.IncludePaths[i], wantIncludes[i])
}
}
wantExcludes := []string{"vendor", "*_generated.go"}
if len(scope.ExcludePatterns) != len(wantExcludes) {
t.Fatalf("ExcludePatterns = %v, want %v", scope.ExcludePatterns, wantExcludes)
}
for i := range wantExcludes {
if scope.ExcludePatterns[i] != wantExcludes[i] {
t.Fatalf("ExcludePatterns[%d] = %q, want %q", i, scope.ExcludePatterns[i], wantExcludes[i])
}
}
}

func stringPtr(value string) *string { return &value }
4 changes: 2 additions & 2 deletions internal/adapters/outbound/reposyncgraph/updater.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ type Updater struct {
Syncer workflow.IncrementalSyncer
}

// Update preserves replace, size limit, namespace, and unreadable-file semantics.
// Update preserves replace, source scope, size limit, namespace, and unreadable-file semantics.
// @intent replace one synchronized repository namespace using the existing incremental ingest contract.
func (u Updater) Update(ctx context.Context, req reposync.GraphRequest) (reposync.UpdateStats, error) {
stats, err := u.Service.Update(requestctx.WithNamespace(ctx, req.Namespace), workflow.UpdateOptions{BuildOptions: workflow.BuildOptions{Dir: req.RepoDir, IncludePaths: req.IncludePaths, MaxFileBytes: req.MaxFileBytes, MaxTotalParsedBytes: req.MaxTotalParsedBytes}, Syncer: u.Syncer, Replace: true, FailOnUnreadable: req.FailOnUnreadable})
stats, err := u.Service.Update(requestctx.WithNamespace(ctx, req.Namespace), workflow.UpdateOptions{BuildOptions: workflow.BuildOptions{Dir: req.RepoDir, IncludePaths: req.IncludePaths, ExcludePatterns: req.ExcludePatterns, MaxFileBytes: req.MaxFileBytes, MaxTotalParsedBytes: req.MaxTotalParsedBytes}, Syncer: u.Syncer, Replace: true, FailOnUnreadable: req.FailOnUnreadable})
if err != nil {
return reposync.UpdateStats{}, err
}
Expand Down
68 changes: 68 additions & 0 deletions internal/adapters/outbound/reposyncgraph/updater_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
package reposyncgraph_test

import (
"context"
"os"
"path/filepath"
"testing"

"github.com/tae2089/code-context-graph/internal/adapters/outbound/reposyncgraph"
"github.com/tae2089/code-context-graph/internal/app/ingest"
"github.com/tae2089/code-context-graph/internal/app/ingest/workflow"
"github.com/tae2089/code-context-graph/internal/app/reposync"
"github.com/tae2089/code-context-graph/internal/domain/graph"
)

type recordingSyncer struct {
files map[string]ingest.FileInfo
}

func (s *recordingSyncer) SyncWithExisting(_ context.Context, files map[string]ingest.FileInfo, _ []string) (*ingest.SyncStats, error) {
s.files = make(map[string]ingest.FileInfo, len(files))
for path, file := range files {
s.files[path] = file
}
return &ingest.SyncStats{Added: len(files)}, nil
}

type sourceParser struct{}

func (sourceParser) Parse(string, []byte) ([]graph.Node, []graph.Edge, error) {
return nil, nil, nil
}

func (sourceParser) ParseWithContext(context.Context, string, []byte) ([]graph.Node, []graph.Edge, error) {
return nil, nil, nil
}

func TestUpdater_ExcludePatternsSkipMatchingFiles(t *testing.T) {
repoDir := t.TempDir()
for name, content := range map[string]string{
"keep.go": "package sample\n",
"skip.gen.go": "package sample\n",
} {
if err := os.WriteFile(filepath.Join(repoDir, name), []byte(content), 0o644); err != nil {
t.Fatalf("write %s: %v", name, err)
}
}

syncer := &recordingSyncer{}
updater := reposyncgraph.Updater{
Service: &workflow.Service{Parsers: map[string]workflow.Parser{".go": sourceParser{}}},
Syncer: syncer,
}

if _, err := updater.Update(context.Background(), reposync.GraphRequest{
RepoDir: repoDir,
Namespace: "api",
ExcludePatterns: []string{"*.gen.go"},
}); err != nil {
t.Fatalf("Update() error = %v", err)
}
if _, ok := syncer.files["keep.go"]; !ok {
t.Fatalf("expected keep.go to be synced, got files=%v", syncer.files)
}
if _, ok := syncer.files["skip.gen.go"]; ok {
t.Fatalf("expected skip.gen.go to be excluded, got files=%v", syncer.files)
}
}
17 changes: 13 additions & 4 deletions internal/app/reposync/ports.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,25 @@ type Checkout interface {
Sync(ctx context.Context, request CheckoutRequest) (repoDir string, err error)
}

// IncludePaths loads repository-local graph include configuration.
// BuildScope captures the repository-local source selection policy for one webhook sync.
// @intent carry include and exclude configuration together so every webhook update uses one coherent build scope.
type BuildScope struct {
IncludePaths []string
ExcludePatterns []string
}

// BuildScopeLoader loads repository-local graph source selection configuration.
// @intent separate repository config file parsing from repository sync orchestration.
type IncludePaths interface {
Load(repoDir string) ([]string, error)
type BuildScopeLoader interface {
Load(repoDir string) (BuildScope, error)
}

// GraphRequest carries the exact webhook update contract to the ingest adapter.
// @intent preserve namespace, replace limits, include paths, and readability policy across the app boundary.
// @intent preserve namespace, source scope, replace limits, and readability policy across the app boundary.
type GraphRequest struct {
RepoDir, Namespace string
IncludePaths []string
ExcludePatterns []string
MaxFileBytes, MaxTotalParsedBytes int64
FailOnUnreadable bool
}
Expand All @@ -40,6 +48,7 @@ type GraphUpdater interface {
// CacheInvalidator drops query results only after a successful graph update.
// @intent keep derived query cache invalidation after successful repository graph commit.
type CacheInvalidator interface{ Invalidate() }

// CacheInvalidatorFunc adapts a runtime closure to the application port.
// @intent adapt runtime cache invalidation closures to the app-owned port.
type CacheInvalidatorFunc func()
Expand Down
Loading
Loading