PySynthData

Synthetic data generation for ML training. Create realistic datasets for model development and testing.

Generate synthetic datasets that match your production data characteristics. Useful for testing, development, and training machine learning models without exposing real data.

Quick Start

from pysynthdata import Generator

# Create synthetic data matching your schema
generator = Generator(
    rows=10000,
    schema={
        "user_id": "integer",
        "email": "email",
        "age": {"type": "integer", "min": 18, "max": 100},
    }
)

df = generator.generate()

Features

Data Generation

• Realistic synthetic data from schema definitions
• Statistical distribution matching (mean, std, quartiles)
• Support for 20+ data types (int, float, string, email, phone, date, etc.)
• Configurable constraints and ranges
• Correlation preservation between fields
• Time series data generation

Data Types Supported

• Numeric: integer, float, decimal (with min/max, distribution)
• Text: string, email, phone, URL, address
• Temporal: date, datetime, time (with ranges)
• Categorical: enum, weighted selection
• Special: UUID, hash, credit card (valid format)
• Relationships: foreign keys, linked data

Distribution Control

• Normal (Gaussian) distribution
• Uniform distribution
• Exponential distribution
• Custom weighted distributions
• Skewness and kurtosis control
• Correlation matrices

Advanced Features

• Seed-based reproducibility
• Field-level correlations
• Referential integrity (foreign keys)
• Missing value patterns
• Data quality simulation (duplicates, outliers)
• Conditional field generation

Export Formats

• Pandas DataFrame
• CSV files
• Parquet (columnar format)
• JSON/JSONL
• SQL INSERT statements

Requirements

• Python 3.10+
• Rust-powered generation engine (precompiled)
• NumPy ≥1.20.0
• Optional: Pandas ≥1.3.0, PyArrow ≥10.0.0

Installation

pip install pysynthdata
# or with uv
uv pip install synthdata

# Verify installation
synthdata --version

Examples

Basic Schema Definition

from pysynthdata import Generator

# Define data schema
schema = {
    "user_id": {"type": "integer", "min": 1000, "max": 999999},
    "email": "email",
    "age": {"type": "integer", "min": 18, "max": 80},
    "country": {"type": "enum", "values": ["US", "UK", "CA", "AU"]},
    "created_at": {"type": "datetime", "min": "2020-01-01"},
}

# Generate dataset
generator = Generator(rows=10000, schema=schema)
df = generator.to_pandas()

print(f"Generated {len(df)} rows matching schema")

Distribution Control

# Match production data distributions
schema = {
    "age": {
        "type": "integer",
        "min": 18,
        "max": 80,
        "distribution": "normal",
        "mean": 45,
        "std": 15,  # Normal distribution, mean=45, std=15
    },
    "income": {
        "type": "float",
        "distribution": "log_normal",
        "mean_log": 11.2,  # Skewed distribution
    },
}

generator = Generator(rows=50000, schema=schema)
df = generator.generate()

# Data matches production statistics
print(f"Age mean: {df['age'].mean():.1f}")  # ~45
print(f"Age std: {df['age'].std():.1f}")    # ~15

Correlation Between Fields

# Create correlated features
schema = {
    "product_price": {"type": "float", "min": 10, "max": 1000},
    "quantity": {"type": "integer", "min": 1, "max": 100},
}

# Correlations: higher price → lower quantity
correlations = {
    ("product_price", "quantity"): -0.7,
}

generator = Generator(
    rows=5000,
    schema=schema,
    correlations=correlations
)

df = generator.to_pandas()

# Verify correlation
print(f"Correlation: {df['product_price'].corr(df['quantity']):.2f}")  # ~-0.7

Time Series Data

# Generate time series data
schema = {
    "timestamp": {"type": "datetime", "start": "2024-01-01", "freq": "1H"},
    "sensor_value": {
        "type": "float",
        "distribution": "normal",
        "mean": 50,
        "std": 5,
    },
    "anomaly": {"type": "boolean", "probability": 0.05},  # 5% anomalies
}

generator = Generator(rows=24*365, schema=schema)  # 1 year of hourly data
df = generator.to_pandas()

Referential Integrity

# Generate related tables
users_schema = {
    "user_id": {"type": "uuid"},
    "name": "string",
    "email": "email",
}

orders_schema = {
    "order_id": {"type": "uuid"},
    "user_id": {"type": "foreign_key", "reference": "user_id"},
    "amount": {"type": "float", "min": 10, "max": 1000},
}

users_gen = Generator(rows=1000, schema=users_schema)
orders_gen = Generator(
    rows=5000,
    schema=orders_schema,
    foreign_keys={"user_id": users_gen}
)

users_df = users_gen.to_pandas()
orders_df = orders_gen.to_pandas()

# Verify referential integrity
assert orders_df['user_id'].isin(users_df['user_id']).all()

Export Options

generator = Generator(rows=100000, schema=schema)

# Export to different formats
generator.to_csv("data.csv")
generator.to_parquet("data.parquet")
generator.to_json("data.jsonl")
generator.to_sql("INSERT INTO users VALUES ...")

df = generator.to_pandas()

Data Quality Simulation

# Simulate real-world data issues
schema = {
    "customer_id": {"type": "integer"},
    "email": {
        "type": "email",
        "missing_percentage": 0.05,  # 5% missing values
    },
    "phone": {
        "type": "phone",
        "missing_percentage": 0.10,  # 10% missing
        "duplicate_percentage": 0.02,  # 2% duplicates
    },
    "amount": {
        "type": "float",
        "outlier_percentage": 0.01,  # 1% outliers
    },
}

generator = Generator(rows=10000, schema=schema)
df = generator.to_pandas()

# Realistic data with imperfections
print(f"Missing emails: {df['email'].isna().sum()}")  # ~500

API Reference

Generator Class

Generator(
    rows: int,
    schema: dict,
    correlations: dict = None,
    foreign_keys: dict = None,
    seed: int = None,
    random_state: int = None
)

Parameters

• rows: Number of rows to generate
• schema: Column definitions with types and constraints
• correlations: Dict of field pairs with correlation values (-1.0 to 1.0)
• foreign_keys: Dict mapping foreign key fields to parent generators
• seed: Random seed for reproducibility
• random_state: NumPy random state (optional)

Methods

• generate() → List[Dict]

  • Returns raw list of records

• to_pandas() → pd.DataFrame

  • Convert to Pandas DataFrame

• to_csv(path) → None

  • Export to CSV file

• to_parquet(path) → None

  • Export to Parquet (columnar)

• to_json(path) → None

  • Export to JSONL format

• to_sql(table_name) → str

  • Generate SQL INSERT statements

Schema Types

# Simple type
"column_name": "email"

# Type with constraints
"column_name": {
    "type": "integer",
    "min": 0,
    "max": 100,
}

# Distribution control
"column_name": {
    "type": "float",
    "distribution": "normal",
    "mean": 50,
    "std": 10,
}

# Missing and quality
"column_name": {
    "type": "string",
    "missing_percentage": 0.05,
    "duplicate_percentage": 0.02,
}

Use Cases

• Testing data pipelines without real data
• ML model training and development (development env isolation)
• Load testing databases and systems
• Privacy-preserving data sharing (synthetic instead of real)
• Benchmarking data processing applications
• Data augmentation for model training

Documentation

• API Reference
• Examples
• Schema Definition

License

MIT License - See LICENSE