feat: realistic graph topology, and metrics to prove it (claude) - #36
Open
denironyx wants to merge 6 commits into
Open
feat: realistic graph topology, and metrics to prove it (claude)#36denironyx wants to merge 6 commits into
denironyx wants to merge 6 commits into
Conversation
TLS certificate verification was disabled for every flight-data download (`verify=False` on both BTS requests, plus a blanket `urllib3.disable_warnings`). That silently exposed all users to man-in-the-middle tampering of the downloaded data. Verification is now on by default. The BTS host has historically served a chain some systems fail to validate, which is presumably why this was turned off, so the escape hatch is preserved as an explicit opt-in: GRAPHFAKER_INSECURE_TLS=1 skips verification and logs a warning saying the data is no longer authenticated. Also declares `tqdm` and `urllib3`, which were imported but absent from `dependencies`. Because `graphfaker/__init__` imports `core`, which imports the flights fetcher, a clean install into an environment without tqdm failed at `import graphfaker`. Bumps version to 0.4.0 and ignores exported *.graphml artifacts. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Duplicate entities are the most frequently reported defect in
LLM-extracted knowledge graphs, and every edge attached to a false node
is a false edge. Tabular record-linkage tools compare *rows*, so they
cannot use the strongest signal a graph offers: two nodes sharing most of
their neighbours are probably the same entity, however differently their
names are spelled.
Adds `graphfaker.resolve`:
- `resolve_entities()` scores candidate pairs on attribute similarity
*and* neighbourhood overlap, then clusters the survivors. The
combination is `attr + w * structural * (1 - attr)`, so structure can
only raise a score and isolated nodes are never penalised for having
few neighbours. `structural_weight=0` degrades to plain attribute
matching.
- `merge_clusters()` collapses each cluster onto one canonical node,
rewiring incident edges, dropping self-loops the merge creates,
counting collapsed parallel edges, and recording both provenance
(`_merged_from`) and conflicting values (`_merged_variants`) so a
merge is never silently lossy.
- `evaluate_clusters()` computes pairwise and B-cubed
precision/recall/F1 against gold labels the caller supplies. It
scores a clustering; it does not manufacture ground truth.
- `GraphFaker.resolve()` as the convenience entrypoint.
Blocking keeps this near-linear; unblocked comparison is guarded against
accidental O(n^2) runs. No new dependencies — string similarity uses
`difflib`. Resolution is deterministic and never mutates the input graph.
Also makes synthetic generation reproducible. `GraphFaker(seed=...)`,
`generate_graph(..., seed=...)`, `reseed()`, and `--seed` now exist, and
generation uses per-instance `random.Random` and `Faker` objects rather
than module-level globals, so seeding no longer perturbs the caller's
global `random` state.
Fixes along the way:
- `cli.py` imported `logger` from the standard library `venv` module,
so all CLI logging went through the wrong logger.
- `generate_graph` reported "Use 'random' or 'osm'" for an unknown
source, naming an option that does not exist and omitting two that
do.
- GraphML export failed on container-valued node attributes; lists,
tuples, sets, and dicts are now flattened.
- Three CLI tests were failing on `main` because their mocks returned
strings where the CLI requires a graph. They now use real graphs, and
tests write exports to tmp_path instead of the repository root.
45 new tests; full suite green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Documents `resolve()`, `evaluate_clusters()`, seeding, and the TLS opt-out in the README, and records everything in HISTORY. Removes three README sections advertising features that do not exist: RDF/JSON-LD/CSV export, Neo4j/Kuzu/TigerGraph integration, million-node scale, and LLM-driven fetching. None shipped in the eleven months since they were written. A "Scope" section now states plainly that anything undocumented is unimplemented. Adds PROPOSAL_V2.md, a strategy assessment that evaluates four candidate directions for the project, records the evidence for dropping each, and documents an eval-focused thesis that was tested and discarded along with the research that killed it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Export
------
Getting a graph out of NetworkX and into an engine has been the sharpest
rough edge. Rather than a driver per database — each needing credentials,
a version matrix, and a live service to test against — this writes files
every engine's own loader already reads:
- export_csv generic node/edge CSV
- export_neo4j_csv `neo4j-admin database import` typed headers
(:ID, :LABEL, :START_ID, :TYPE, :END_ID)
- export_cypher Neo4j, openCypher, and ISO GQL dialects
Also wired to the CLI as --format. CSV covers TigerGraph LOAD, Neptune's
bulk loader, and Spark/GraphFrames; once loaded, Neo4j GDS runs directly
on the result.
CSV headers are the *union* of every key seen, not the first node's keys.
Taking the first node's keys silently corrupts heterogeneous graphs: a
Person header gets written and then every Place's values land under it,
shifted. Container values are flattened, non-ASCII is written as UTF-8
rather than the platform default, empty graphs do not raise, and
multigraph edges are preserved.
Corpus
------
`graphfaker.corpus` builds documents whose entities are known in advance,
so the number of nodes a builder creates per real entity can be counted.
Nothing is corrupted. The text is clean, well-formed English and every
entity is unambiguous to a human reader, so a correct pipeline scores
zero. Entities are named with the surface forms a writer naturally uses —
surname alone, an accepted abbreviation — which is ordinary prose rather
than injected noise. This restraint is the point: synthetic corruption is
known to be far easier than real error (Lam et al., IJPDS 2024, roughly a
hundredfold gap), so a benchmark built on guessed error rates measures
its own noise model. Counting splits of entities a human would never
split is a weaker claim that a generator can actually support.
Two safeguards, both learned from getting it wrong first:
- Names are vetted for mutual distinctness. Naming four people
"Person 0".."Person 3" yields strings ~93% similar, so any matcher
merges them and the "duplication" is the corpus's fault.
- Aliases are assigned in a second pass, once every name is known, and
rejected when another entity could claim them. Faker derives place
names from surnames, so a person aliased "Henderson" beside an org
named "Henderson, Ramirez and Lewis" is a real collision.
`Corpus.audit()` reports any residual ambiguity and the example harness
refuses to run unless it is clean.
Also fixes a silent failure in the Cypher exporter: identifier sanitising
stripped the leading underscore, so nodes carried `gf_id` while the
relationship MATCH clauses looked up `_gf_id`. Every relationship failed
to create while the script still ran without error.
Adds requires-python (>=3.10). 45 more tests; suite at 102.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Cognee adapter
--------------
Rewritten against the API as it actually exists in cognee 1.4.1, checked by
introspection rather than assumed: `add`, `cognify`, and `export` are all
coroutines, `export` accepts format="graphml", and there is no
`get_graph_data` — the previous stub called it, and it would have failed on
the first run.
Two deliberate choices:
- Writes to a run-specific dataset instead of calling `cognee.prune`.
Pruning wipes the user's local cognee store, and `export` is scoped
to a dataset, so isolation costs nothing.
- data_cache=False, so a re-run genuinely re-extracts rather than
replaying the previous run and looking falsely stable.
Cognee's graph also holds DocumentChunk/TextSummary/TextDocument
bookkeeping nodes, so the adapter filters to entity labels — and prints
the label distribution it saw, because that schema is not guaranteed
stable and a filter matching nothing would otherwise report as "100% of
entities missed" rather than as a wrong filter.
Note: cognee requires an LLM API key even for add(), which runs a
pipeline that tests the LLM connection before ingesting. The API surface
here is verified; the end-to-end output shape is not, and the notebook
inspects it at runtime rather than assuming.
Notebook
--------
docs/notebooks/duplication_experiment.ipynb runs the whole experiment:
build a corpus, audit it, run cognee, inspect what got split, repair with
resolve(), sweep the threshold, export the result. It executes clean
without an LLM key using a clearly-labelled simulated extraction, so it
works as a tutorial.
Name-variant matching
---------------------
The notebook exposed a real weakness. "Hill" against "Allison Hill"
scores ~0.5 on character ratio, because most of the longer string is
unmatched — so with any useful threshold the pair was discarded before
neighbourhood overlap was ever consulted. Referring back to an entity by
a shorter form is one of the commonest things a document does, so this
was a large recall hole: on the notebook's graph, resolve() was fixing
nothing at all.
`token_subset_floor` (0.75, on by default) floors the attribute score
when one name's tokens are contained in the other's. It floors rather
than sets to 1.0 on purpose: containment is suggestive, not conclusive —
"Smith" may be a different person from "John Smith" — so it lifts the
pair into consideration and leaves the decision to shared neighbours.
Single-letter tokens are excluded so initials are not treated as names.
Effect on the notebook's simulated graph: duplication 60% -> 20%, node
inflation 2.15x -> 1.20x, at pairwise precision 1.000 and recall 0.842.
A limitation worth stating: structural matching only helps when duplicate
nodes share neighbours. An extractor that partitions an entity's edges
across its duplicates leaves almost no overlap, and that is the hard
case. The notebook says so and sweeps the threshold instead of implying
one setting is correct.
Also fixes an alias with dangling punctuation — the first word of
"Barnes, Cole and Ramirez" is "Barnes," and aliases appear verbatim in
the generated prose.
Suite at 114.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both endpoints of every edge were drawn uniformly at random, so the
synthetic generator produced an Erdos-Renyi graph: Poisson degree
distribution, no hubs, effectively no clustering, no community structure,
and attributes statistically independent of the topology. It was
described as realistic and was not, and anything measured against it was
measured against noise.
Edges are now formed by three mechanisms, each addressing one missing
property:
- preferential attachment, via the repeated-entry trick so sampling
stays O(1) per edge rather than rebuilding a weight vector
- triadic closure, which is the entire source of clustering
- homophily over latent communities, with candidates drawn mostly from
the source's own community — affinity scoring alone left modularity
near zero, because a global sample of 8 usually contains no
same-community candidate to prefer
Measured on 600 nodes / 2400 edges, seed 1:
metric realistic uniform (before)
degree Fano factor 7.9 1.7
max degree 88 19
degree Gini 0.45 0.26
average clustering 0.180 0.013
community modularity 0.72 -0.004
age homophily 0.81 0.01
isolated nodes 0 4
The Fano factor is the clearest single test: a Poisson distribution has
variance equal to its mean, so uniform attachment sits near 1 by
construction.
Adds graphfaker.metrics (graph_stats, compare_topology) so the claim is
checkable rather than asserted. The new tests are differential — every
property is compared against topology="uniform", which reproduces the old
behaviour and is retained solely as a baseline. One test asserts the
baseline still exhibits the old defect, because the comparison would be
meaningless if it had also been fixed.
Attributes now agree with structure:
- population tracks connectivity; employee_count correlates 0.95 with
the WORKS_AT edges actually present instead of contradicting them
- ages cluster by community, which is what makes homophily measurable
- industry holds an industry, not fake.job()'s job titles
- LIVES_IN, BORN_IN and HEADQUARTERED_IN are singular; a person could
previously live in four cities
Two contract fixes found while measuring:
- total_edges now means what number_of_edges() reports. Counting loop
iterations overshot ~14% when bidirectional friendships dominated and
undershot ~12% when skipped functional relationships did.
- No isolated nodes. Preferential attachment alone left 24% of a
600-node graph unreachable, so coverage and top-up passes guarantee a
giant component.
Also fixes GraphML export raising on None attribute values, which broke
export for any graph small enough to leave a community without a place,
and pins the Wikipedia test — it called the live API and failed
intermittently. The live check is preserved behind a `network` marker.
Worth knowing when comparing across versions: entity resolution is
measurably harder on a realistic graph. The same injected duplicates give
resolve() precision 1.000 on a uniform graph and 0.88 on a realistic one,
because homophily means distinct people share attributes and neighbours.
Numbers produced before this change were flattered by the generator.
Suite at 144 (143 offline plus one network-marked).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prompted Opus 5 to come up with a new release and feature plan.