diff --git a/techy/01-data-science.md b/techy/01-data-science.md new file mode 100644 index 0000000..ed4f143 --- /dev/null +++ b/techy/01-data-science.md @@ -0,0 +1,609 @@ +# Data Science — Interview Preparation + +--- + +## Data Engineering vs Data Science + +**Definition:** Data Engineering builds the infrastructure to collect, store, and process data at scale; Data Science analyzes that data to extract insights and build models. + +### Theory & Explanation + +Data Engineering and Data Science are complementary but distinct disciplines. Data Engineering focuses on the architecture, pipelines, and infrastructure that enable data to flow reliably from source systems to storage and compute platforms. Data Engineers build and maintain ETL/ELT pipelines, design data warehouses and data lakes, manage orchestration (Airflow, Prefect), and ensure data quality, scalability, and operational reliability. Their primary output is trustworthy, well-structured data delivered on schedule. + +Data Science focuses on extracting insight from data through statistical analysis, hypothesis testing, machine learning, and visualization. Data Scientists explore data, formulate and test hypotheses, build predictive models, design experiments (A/B tests), and communicate findings to stakeholders. Their primary output is analytical insights, models, and data-driven recommendations. While there is overlap in tools (Python, SQL), the orientation differs fundamentally: Data Engineering is systems engineering focused on reliability and scale; Data Science is analytical discovery focused on inference and prediction. + +The data science hierarchy of needs makes this relationship explicit: data engineering (infrastructure, pipelines, storage) forms the foundation — without reliable data collection, ingestion, and transformation, no amount of sophisticated modeling produces trustworthy results. A team with excellent data scientists but poor data engineering will struggle with unreliable data, slow iteration cycles, and distrust in model outputs. + +| Dimension | Data Engineering | Data Science | +|-----------|-----------------|--------------| +| Primary focus | Infrastructure, pipelines, reliability | Analysis, modeling, inference | +| Key output | Clean, reliable data at scale | Insights, models, recommendations | +| Core skills | Distributed systems, SQL, orchestration, cloud infra | Statistics, ML, experimentation, visualization | +| Typical tools | Airflow, Spark, dbt, Snowflake, Kafka | Python/R, scikit-learn, Jupyter, TensorFlow | +| Stakeholders | Analytics teams, downstream consumers | Business stakeholders, product teams | + +### Example + +A Data Engineer builds a pipeline that streams clickstream events from a mobile app through Kafka, buffers them in S3, runs Spark transformations for sessionization and enrichment, and loads the result into Redshift. The pipeline runs hourly with automated retry logic, data quality checks, and freshness alerts. A Data Scientist then queries the Redshift tables to build a churn prediction model: they engineer features from the clickstream data (session frequency, time-on-page, feature usage), train an XGBoost classifier, evaluate with precision-recall, and deploy the model to score users daily. + +### Interview Questions + +**Q: Can a data scientist do data engineering?** +A: Data scientists often handle light data engineering (SQL transformations, basic pipelines), but production-grade data engineering requires deep knowledge of distributed computing, cloud infrastructure, orchestration, error handling, and operational monitoring. A data scientist performing DE tasks at scale typically leads to fragile pipelines, poor performance, and unreliable data delivery. The ideal is collaboration — the DE builds reliable infrastructure that enables the DS to focus on analysis. + +**Q: What happens if data engineering infrastructure is poor?** +A: Poor DE infrastructure causes cascading failures: data arrives late or not at all, pipelines break silently, data quality degrades, and dashboards show stale or incorrect numbers. Data scientists waste 60-80% of their time on data cleaning and wrangling instead of analysis. Models trained on unreliable data produce incorrect predictions, eroding stakeholder trust. The root cause is almost never the modeling — it is the data pipeline. + +--- + +## Data Warehousing + +**Definition:** Centralized repository for structured, cleansed, integrated data from multiple sources, optimized for query and analysis. + +### Theory & Explanation + +Data warehousing follows two major architectural philosophies. Kimball (dimensional modeling) advocates a bottom-up approach: build dimensional data marts for individual business processes first, then integrate them into a conformed dimension bus architecture. The core modeling pattern is the **star schema** — a central **fact table** containing quantitative measures and foreign keys to surrounding **dimension tables** with descriptive attributes. Inmon advocates a top-down approach: build a normalized enterprise data warehouse in 3NF first, then derive departmental data marts. Kimball is faster to deliver and more intuitive for business users; Inmon provides a single source of truth but takes longer to build. + +**OLAP vs OLTP** is a fundamental distinction. OLTP (Online Transaction Processing) systems handle high-volume, low-latency transactions with row-oriented storage and ACID guarantees — designed for inserts and updates. OLAP (Online Analytical Processing) systems handle complex aggregations over large datasets with column-oriented storage and query optimization — designed for reads and analysis. A data warehouse is an OLAP system. + +**Slowly Changing Dimensions (SCDs)** manage how dimension attributes change over time. Type 1 overwrites the old value with the new (no history). Type 2 creates a new row with effective date ranges, preserving full history. Type 3 adds a new column to track the previous value (limited history). Type 2 is most common in analytics. The data warehouse vs data lake vs data lakehouse distinction: data warehouses offer best query performance and governance but require structured schemas; data lakes store raw data cheaply but lack ACID and performance; data lakehouses combine the two with ACID transactions on object storage (Databricks Delta Lake, Iceberg, Hudi). + +### Example + +A sales data warehouse includes `fact_sales` with columns `sale_id`, `date_key`, `customer_key`, `product_key`, `store_key`, `quantity`, and `amount`. Dimension tables: `dim_customer` (name, address, segment), `dim_product` (name, category, brand, price), `dim_date` (date, day_of_week, month, quarter, year), `dim_store` (name, region, size_sqft). A typical analytical query: "What were total sales by product category per month in Q4 2024?" — joins fact_sales → dim_product → dim_date, groups by category and month, aggregates sum(amount). + +### Interview Questions + +**Q: Star schema vs snowflake schema — what are the trade-offs?** +A: Star schemas denormalize dimensions into a single table per dimension — simpler, faster queries with fewer joins, but more storage due to redundancy. Snowflake schemas normalize dimensions into sub-dimensions — less storage, better referential integrity, but more joins and slower query performance. Star is preferred for analytics due to simpler query patterns; snowflake is sometimes used when storage costs dominate or when dimensions are deeply hierarchical. + +**Q: When would you denormalize a data warehouse?** +A: Denormalization is warranted when query performance is critical and the redundancy cost is acceptable — for example, storing product category name directly in the fact table to avoid a join in every query. It is also common in reporting marts where business users need a flat, wide table they can query directly in BI tools without understanding join semantics. + +--- + +## ETL vs ELT + +**Definition:** ETL (Extract, Transform, Load) transforms data before loading; ELT (Extract, Load, Transform) loads raw data first, transforms in the target system. + +### Theory & Explanation + +ETL is the traditional approach. Data is extracted from source systems, transformed in a staging area (cleansing, validation, deduplication, aggregation, PII removal), and then loaded into the target data warehouse. The transformation step occurs before loading, so the warehouse receives clean, conformed data. ETL provides better **data quality and compliance** because sensitive data (PII, PHI) can be removed or masked before it enters the warehouse. However, ETL is slower (transformation creates a bottleneck), less flexible (transforms are pre-defined and hard to retroactively change), and requires a separate transformation engine (Informatica, Talend, SSIS). + +ELT is the modern cloud-native approach made possible by massively parallel processing (MPP) data warehouses (Snowflake, Redshift, BigQuery). Raw data is extracted and loaded directly into the warehouse with minimal transformation. Transformations are applied later using the warehouse's compute engine (SQL, dbt). ELT is **faster** (loading raw data is quick), **more flexible** (raw data is available for reprocessing or new analysis), and **leverages MPP compute** rather than requiring a separate transformation server. ELT is preferred for agile analytics, large data volumes, frequent schema changes, and scenarios where the raw data has ongoing analytical value. + +The tool landscape reflects this split. dbt is the dominant ELT transformation tool — analysts write SQL SELECT statements that are materialized as views or tables. Fivetran and Airbyte handle the EL part (extract and load). Traditional ETL tools like Informatica and Talend remain relevant in regulated industries where data must be cleansed before entering any storage. + +### Example + +A company handling EU customer data uses **ETL for GDPR compliance**: customer names and email addresses are pseudonymized in the staging layer before any data reaches the warehouse. This ensures that even if the warehouse is compromised, raw PII is never exposed. For **advertising analytics**, the same company uses **ELT**: raw clickstream events are loaded into Snowflake as-is, then analysts use dbt to define sessionization, attribution, and aggregation transforms. When a new attribution model is needed, analysts can rebuild transforms against the raw data without re-extracting from source. + +### Interview Questions + +**Q: Which is better for data quality?** +A: ETL has inherently stronger data quality guarantees because transformation and validation happen before data enters the warehouse — bad data can be rejected, quarantined, or corrected at the staging layer. ELT shifts this responsibility downstream: data quality checks are applied during transformation within the warehouse, which means raw data with quality issues may be loaded and could be consumed by downstream processes before quality checks run. Modern ELT pipelines address this with automated quality testing (dbt tests, Great Expectations) running immediately after load. + +**Q: When would you use ETL despite ELT being faster?** +A: Use ETL when: (1) strict regulatory compliance requires data to be anonymized before storage; (2) source systems have limited bandwidth and need transformation to reduce data volume before transfer; (3) the target warehouse charges significant compute costs and pre-transforming reduces storage/processing cost; (4) the transformation engine has capabilities the warehouse cannot match (complex streaming transforms, proprietary libraries); (5) legacy infrastructure predates modern cloud data warehouses. + +--- + +## Data Modeling (Star Schema, Snowflake Schema) + +**Definition:** Process of defining how data is structured, connected, and stored. Key schemas for analytics: Star and Snowflake. + +### Theory & Explanation + +Star Schema is the foundational dimensional modeling pattern. A central **fact table** stores quantitative measures (sales amount, quantity, count) and foreign keys to surrounding **dimension tables**. Dimension tables are **denormalized** — all descriptive attributes for an entity live in a single table. This design makes the schema simple (resembling a star) and queries fast because they require only single-hop joins from fact to dimension. Star schemas are intuitive for business users and perform well in OLAP engines. + +Snowflake Schema extends star by **normalizing** dimension tables into sub-dimensions. For example, `dim_product` (product_id, product_name, category_id) joins to `dim_category` (category_id, category_name, department_id), which joins to `dim_department` (department_id, department_name). This reduces data redundancy and improves referential integrity but increases the number of joins required for queries. Snowflake schemas use less storage but can degrade query performance on large datasets. + +| Dimension | Star Schema | Snowflake Schema | +|-----------|-------------|------------------| +| Complexity | Simple, single join per dimension | Multiple joins per dimension | +| Query speed | Faster (fewer joins) | Slower (more joins) | +| Storage | Higher (redundant attributes) | Lower (normalized) | +| Maintenance | Update anomalies possible | Better integrity | +| Business user | Easy to understand | More complex | + +**Dimensional modeling** differs from **3NF** (Third Normal Form). 3NF removes all redundancy through normalization — it is designed for transactional systems with high write volumes. Dimensional modeling accepts controlled redundancy for query performance and usability. **Slowly changing dimensions** manage how dimension attributes evolve: Type 1 (overwrite), Type 2 (new row with date range), Type 3 (new column for previous value). Type 2 is most common in analytics for accurate historical reporting. + +### Example + +Star Schema: `dim_product` contains `product_id`, `product_name`, `category_id`, `category_name`, `brand_name` — all in one table. Query: `SELECT SUM(amount) FROM fact_sales JOIN dim_product ON ...` — one join. + +Snowflake Schema: `dim_product` contains `product_id`, `product_name`, `category_id`; `dim_category` contains `category_id`, `category_name`, `brand_id`. Query: `SELECT SUM(amount) FROM fact_sales JOIN dim_product ON ... JOIN dim_category ON ... JOIN dim_brand ON ...` — three joins. + +### Interview Questions + +**Q: Why would you use snowflake over star?** +A: Snowflake is useful when: (1) storage costs are a primary concern (normalization reduces data volume); (2) dimensions have deep hierarchies that are themselves analyzed independently; (3) you need strict referential integrity enforced by the schema; (4) multiple dimensions share sub-dimensions (e.g., customer and employee both reference geography). In practice, most production data warehouses use star schemas because query performance matters more than storage savings in modern cloud environments. + +**Q: What is a degenerate dimension?** +A: A degenerate dimension is a dimension key stored in the fact table with no corresponding dimension table. It occurs when the dimension attribute has no other descriptive attributes — for example, a transaction ID or order number. These are stored directly in the fact table as a single column rather than creating a dimension table with only one column. Common in transactional fact tables. + +--- + +## Data Lake + +**Definition:** Centralized repository for storing raw data in native format (structured, semi-structured, unstructured) at any scale. + +### Theory & Explanation + +A data lake stores data **as-is**, with no schema required at write time — the schema is applied at read time (schema-on-read). This is the fundamental difference from a data warehouse, which requires schema-on-write. Data lakes are built on inexpensive object storage (Amazon S3, Azure Data Lake Storage, Google Cloud Storage) and can hold any data type: structured tables (CSV, Parquet), semi-structured data (JSON, XML, Avro), unstructured data (images, video, audio, PDFs, binary files). Storage costs are an order of magnitude cheaper than data warehouses. + +The key advantages are **massive scale** (exabytes), **low cost**, and **schema flexibility** — new data sources can be ingested without schema design, and raw data is preserved for future use cases not anticipated at ingestion time. However, data lakes without governance quickly degenerate into **data swamps** — unorganized, undocumented repositories where data is impossible to find or trust. Challenges include poor data quality, lack of ACID transactions (partial writes during failures leave inconsistent state), performance limitations for interactive queries, and difficulty enforcing access controls at the record level. + +**Delta Lake, Apache Iceberg, and Apache Hudi** are table formats that address these limitations by adding ACID transactions, time travel (versioning), schema enforcement, and efficient upserts to data lakes — creating a **data lakehouse** architecture. The lakehouse combines the flexibility and low cost of data lakes with the reliability and performance of data warehouses. + +### Example + +A company ingests mobile app clickstream data as raw JSON logs into an S3 data lake — each event file is timestamped and partitioned by date. A Spark job reads the raw JSON, performs schema inference, deduplication, and sessionization, then writes structured Parquet files back to the lake's cleaned zone. From there, structured data can be loaded into Redshift for BI dashboarding, or queried directly with Amazon Athena for ad-hoc analysis. Separately, product images uploaded by users are stored in the same data lake — a data scientist spawns a GPU cluster to run feature extraction via a CNN, saving the embedding vectors back to the lake for a visual search model. + +### Interview Questions + +**Q: How do you prevent a data lake from becoming a data swamp?** +A: Prevention requires implementing the same governance practices as a data warehouse: (1) define a clear zone structure (raw → staged → curated/analytics); (2) enforce metadata registration — every dataset must have a catalog entry with owner, description, schema, freshness; (3) implement access controls and data classification; (4) run automated quality checks (schema validation, completeness, freshness); (5) use table formats (Delta Lake, Iceberg) for ACID and versioning; (6) document data lineage via tools like DataHub or Marquez. + +**Q: When to use data lake vs data warehouse?** +A: Use a data lake when: (1) storing raw/unstructured data; (2) the use case is not fully defined (flexibility needed); (3) ML/AI training requires large-scale raw data access; (4) storage costs must be minimized. Use a data warehouse when: (1) BI dashboards and reporting need fast, consistent query performance; (2) business users need direct SQL access; (3) data requires strict schema enforcement; (4) regulatory compliance requires data governance features (audit trails, access control). Increasingly, organizations use a **lakehouse** to get the best of both. + +--- + +## Data Pipeline + +**Definition:** Automated series of steps that ingest, process, transform, and deliver data from source to destination. + +### Theory & Explanation + +A data pipeline follows a lifecycle: **source** → **ingestion** → **processing** → **destination**. The source can be any data producer (transactional database, API, streaming events, file upload). Ingestion is either **batch** (scheduled, processes data in chunks — hourly/daily extracts) or **streaming** (continuous, millisecond latency). Processing includes cleansing, validation, deduplication, type casting, enrichment, aggregation, and business logic transformation. The destination is typically a data warehouse, data lake, analytics engine, or reverse ETL target (CRM, marketing platform). + +**Batch pipelines** are simpler to build and debug: they process finite data on a schedule, load full or incremental snapshots, and retry on failure. **Streaming pipelines** handle unbounded data with stateful processing (windowing, watermarking) and require exactly-once or at-least-once semantics. The choice depends on latency requirements: batch for daily/hourly reporting; streaming for real-time dashboards, fraud detection, or operational alerts. + +**Orchestration** is the backbone of data pipelines. Apache Airflow is the industry standard — pipelines are defined as Directed Acyclic Graphs (DAGs) of tasks with dependencies, retry logic, alerting, and execution history. Alternatives include Prefect (Python-native, easier to use) and Dagster (data-aware, with built-in asset management). Pipeline monitoring tracks data freshness (is today's data loaded?), volume anomalies (sudden drop/increase in row count), schema changes, task failures, and SLA compliance. + +### Example + +An Airflow DAG runs daily: (1) Extract orders from PostgreSQL since last run (incremental load); (2) Validate schema against expected structure (fail if column missing or type mismatch); (3) Transform — clean NULL values, calculate derived columns (total = quantity × price), aggregate customer-level daily metrics; (4) Load to Redshift `fact_daily_orders`; (5) Trigger downstream dbt run to refresh the reporting mart; (6) Send Slack notification with row count and job duration. Each step has retry logic (3 retries, exponential backoff), and if the entire DAG fails, an on-call engineer receives a PagerDuty alert within 5 minutes. + +### Interview Questions + +**Q: Batch vs streaming — how to choose?** +A: Batch is appropriate when: latency requirements are minutes/hours/days; data volume per batch is manageable; sources don't support streaming; the cost of streaming infrastructure isn't justified. Streaming is necessary when: sub-second latency is required (fraud detection, real-time personalization); data is a continuous unbounded stream; the use case requires stateful processing over event sequences. Many organizations start with batch and add streaming incrementally for specific high-value use cases. + +**Q: How do you handle pipeline failures?** +A: A robust failure handling strategy includes: (1) automatic retries with exponential backoff for transient failures (network timeouts, resource contention); (2) idempotent pipeline design (running the same pipeline twice produces the same result); (3) incremental loading with watermark tracking to avoid full rebuilds on failure; (4) dead letter queues for records that repeatedly fail; (5) comprehensive alerting — failures first page on-call within minutes, then escalate; (6) post-mortem analysis — every production failure triggers root cause investigation and prevention measures. + +--- + +## Data Governance + +**Definition:** Overall management framework for data availability, usability, integrity, and security. + +### Theory & Explanation + +Data governance encompasses multiple **pillars**: **Data quality** ensures accuracy, completeness, timeliness, consistency, validity, and uniqueness. **Data stewardship** assigns ownership and accountability for data assets — every dataset has a designated steward responsible for its quality, documentation, and access approvals. **Data security** implements access controls (RBAC, column-level security), encryption at rest and in transit, dynamic data masking, and audit logging. **Data privacy** ensures compliance with regulations (GDPR, CCPA, LGPD) through PII classification, consent management, data retention, and right-to-erasure mechanisms. **Data lifecycle management** defines retention periods, archival strategies, and permanent deletion schedules. **Data standards** establish naming conventions, metadata standards, and documentation requirements across the organization. + +A **governance council** typically includes business and IT representatives who meet regularly to review data assets, approve policies, and resolve disputes. Successful governance requires executive sponsorship — without top-down support, governance becomes an impediment that teams circumvent. + +**Data contracts** are formal agreements between data producers and consumers specifying schema, freshness SLAs, quality guarantees, and breaking change notification periods. They shift governance left — producers commit to not breaking downstream consumers without notice. Tools: Collibra, Alation, Atlan, Databricks Unity Catalog. + +### Example + +A company implements customer data governance: PII columns (name, email, phone) are encrypted at rest with AES-256 and access is logged for auditing. Quarterly reviews check that only authorized teams access customer data. Data retention follows GDPR — customer data is retained for 7 years after account closure, then permanently deleted. The data steward for customer data is the VP of Marketing. Quality SLOs require < 1% null values on email and < 0.1% duplicate customer records. A data contract between the billing team (producer) and the analytics team (consumer) specifies that the `customer_accounts` table will refresh within 2 hours of event time and breaking schema changes require 30 days notice. + +### Interview Questions + +**Q: How would you implement data governance in a company with no existing practices?** +A: Start small and iterate: (1) Identify the 5-10 most critical data assets (revenue data, customer data, regulatory reports); (2) Assign a data steward for each critical asset; (3) Document metadata — schema, owner, description, freshness, quality checks; (4) Implement basic access controls on those critical assets; (5) Establish a data quality monitoring framework (Great Expectations); (6) Create a lightweight governance council with monthly meetings; (7) Define and publish data standards as the team grows. Avoid trying to govern everything at once — it will be overwhelming and resisted. + +**Q: GDPR right to be forgotten — how do you implement it in a data warehouse?** +A: Implementation requires: (1) Identifying all tables containing PII across the warehouse, catalog, data lake, and backups; (2) Implementing a cascading deletion or anonymization process — when a deletion request is received, SQL scripts update/delete PII columns across all tables; (3) Handling Type 2 SCDs — for historical dimension rows, PII must be masked or replaced with a token; (4) Handling backups — you must either restore and scrub or rotate backup encryption keys so old backups are unrecoverable; (5) Handling data lake files — rewrite Parquet/JSON files in the affected partitions; (6) Proving compliance — each deletion must be logged with timestamp and affected records. + +--- + +## Data Catalog + +**Definition:** Organized inventory of data assets with metadata, lineage, and documentation to make data discoverable and understandable. + +### Theory & Explanation + +A data catalog stores three layers of metadata. **Technical metadata** describes the physical data asset: schema (column names, data types, partitions, file format), row count, storage location, access patterns, and freshness. **Business metadata** provides meaning: business definitions, KPI calculations, governed metric definitions, certification status, and usage context. **Operational metadata** captures environment information: data owner, steward, creation date, update frequency, source system, pipeline dependencies, and quality scores. + +The catalog enables **data discovery** — analysts and data scientists search and browse assets by domain, tag, or keyword. Catalogs support social features: ratings, reviews, comments, and popularity signals that help users find the most trusted datasets. **Active metadata** extends this with machine learning — the catalog auto-suggests relationships, flags deprecated tables, recommends popular datasets, and alerts about breaking changes or quality degradation. + +Catalog integration spans the entire data stack: data warehouses, data lakes, BI tools (Looker, Tableau), transformation tools (dbt), and orchestration tools (Airflow). **Open source** options include DataHub (LinkedIn) and Amundsen (Lyft). **Commercial** leaders include Atlan, Alation, and Collibra. + +### Example + +An analyst searches "customer revenue" in the data catalog. The search returns `fact_customer_revenue_monthly` as the top result — it is certified (green badge), well-documented, and highly rated. The catalog entry shows: description ("Revenue by customer per month from the billing system, refreshed daily"), owner ("Sarah J. — Finance"), data source lineage (billing system → Airflow ingestion → Snowflake staging → dbt transforms → Redshift curated), quality score (95%), schemas (columns: customer_id, revenue_month, total_revenue, order_count), and usage analytics (queried 47 times this week, used in 3 Looker dashboards). + +### Interview Questions + +**Q: How does a data catalog differ from a data dictionary?** +A: A data dictionary is a static document (often a spreadsheet or wiki page) listing tables, columns, and their definitions — typically created once and becomes stale. A data catalog is a dynamic, searchable platform that automatically syncs with the data warehouse, tracks lineage, captures usage, enables collaboration, and constantly updates as schemas change. A catalog subsumes the dictionary and adds discovery, governance, and operational features. + +**Q: What information should a data catalog entry contain?** +A: A complete entry should include: (1) Basic identification (name, location, data source type); (2) Technical metadata (schema with column types, row count, partitions, update frequency); (3) Business metadata (description, business owner, definitions of key terms); (4) Quality metadata (completeness %, accuracy checks, certification status); (5) Lineage (upstream sources, downstream transformations and dashboards); (6) Usage (popularity, recent queries, consuming teams); (7) Governance (access requirements, sensitivity classification, retention policy); (8) Collaboration (steward contact, comments, documentation links). + +--- + +## Data Lineage + +**Definition:** Tracking the flow of data from its origin through transformations to its final destination — showing where data comes from, how it changes, and what depends on it. + +### Theory & Explanation + +Data lineage provides end-to-end visibility of data movement: **source system** → **ingestion** → **staging** → **transformation** → **analytics layer** → **dashboard/report**. This enables three critical capabilities. **Impact analysis**: before changing a source table schema, an engineer can see all downstream dependencies — which transformations will break, which dashboards will fail, which models need retraining. **Root cause analysis**: when a dashboard number looks wrong, lineage traces back through every transformation to identify where the error was introduced. **Auditing and compliance**: regulators require proof of data provenance — lineage provides the documented path from raw data to regulated reports. + +**Column-level lineage** is the most granular and valuable form — it tracks individual fields through JOINs, aggregations, CASE statements, and derived calculations. For example, knowing that `customer_lifetime_value` in a dashboard originates from `SUM(order_amount * 0.95)` in a dbt model, which reads `order_amount` from the `stg_orders` table, which extracts it from the raw orders API. + +**Backward lineage** answers "where did this come from?" — tracing upstream. **Forward lineage** answers "what depends on this?" — tracing downstream. **Automated lineage** is generated by parsing transformation code (SQL, Python). dbt automatically generates lineage by parsing `ref()` and `source()` calls in SQL models. For complex Python transforms, manual annotation or deep SQL parsing may be needed. + +### Example + +The CFO asks: "Why is Q3 revenue showing $2M less than expected?" The lineage tool traces from the revenue dashboard back: dashboard reads from `fct_revenue` → which is built in dbt by joining `stg_orders` with `dim_products` → `stg_orders` reads from `raw_orders` with a filter `WHERE order_id IS NOT NULL`. Investigation reveals the filter removed 10% of orders where `order_id` was NULL — but those were valid marketplace orders where the ID was stored in a different column. The fix: update the `stg_orders` transform to handle the alternative ID column. Without lineage, finding this root cause would require manual tracing through a dozen potential failure points. + +### Interview Questions + +**Q: Why is column-level lineage important?** +A: Table-level lineage tells you that a dashboard depends on a table. Column-level lineage tells you exactly which columns flow through which joins and transformations to produce a specific metric. This granularity is essential for: (1) precise impact analysis (a change to `customer_zip_code` won't affect `total_revenue`, but a change to `order_amount` will); (2) debugging (trace a specific number through the pipeline); (3) regulatory compliance (prove that a reporting metric uses the exact approved data sources and calculations). + +**Q: How does dbt generate lineage automatically?** +A: dbt parses `ref()` and `source()` macros in SQL model files to build a dependency graph. Each model file declares its upstream dependencies: `{{ ref('stg_orders') }}` creates an edge from `stg_orders` to the current model. dbt also parses column selections and JOIN conditions to infer column-level lineage, though precise column tracing requires the query compiler (for full SQL parsing). The lineage graph is exposed in the dbt docs website and can be exported to external catalog tools via the manifest.json artifact. + +--- + +## Data Quality + +**Definition:** Measure of data fitness for its intended use, assessed across dimensions including accuracy, completeness, consistency, timeliness, validity, and uniqueness. + +### Theory & Explanation + +Data quality is measured across **six dimensions**. **Accuracy**: data reflects the real-world object or event it represents. **Completeness**: all expected data is present (no missing required fields). **Consistency**: data is coherent across systems (same customer name in CRM and billing). **Timeliness**: data is current enough for its intended use (real-time fraud detection needs sub-second freshness; monthly reporting can tolerate daily updates). **Validity**: data conforms to defined formats, schemas, and business rules (revenue must be ≥ 0, email must contain @). **Uniqueness**: no unintended duplicate records exist. + +**Measurement** requires specific metrics per dimension: completeness = percentage of non-null values for required fields; uniqueness = distinct count / total count for ID fields; timeliness = 90th percentile of ingestion latency; accuracy = sampled records verified against source of truth. **Data quality SLAs** define acceptable thresholds: "Orders must have < 0.1% duplicate records, be loaded within 2 hours of event time, and have 99.9% completeness on required fields." + +**Great Expectations** is the leading open-source data quality framework. You define **expectations** as Python code — `expect_column_values_to_not_be_null('email')`, `expect_column_values_to_be_between('age', 0, 120)`. Expectations are run as pipeline tests and produce data documentation with pass/fail results per run. Quality data is itself stored for trend analysis (is quality improving or degrading?). + +### Example + +A customer data quality audit reveals: completeness is 100% for email (required) but only 60% for phone (optional, unpopulated in 40% of profiles). Validity shows 2% of zip codes don't match the city/state mapping (likely data entry errors or outdated ZIP-to-city mappings). Uniqueness shows 0.5% duplicate customer records (same email address associated with different customer IDs, likely from different signup flows merging incorrectly). Each issue is documented in the data quality dashboard with severity ranking, steward assignment, and a remediation timeline. + +### Interview Questions + +**Q: How would you implement data quality monitoring at scale?** +A: A scalable implementation has three layers. Layer 1 — **pipeline-level checks**: each pipeline step validates data before passing to the next stage (row count ranges, null thresholds, schema validation). Layer 2 — **scheduled monitoring**: Great Expectations runs hourly against critical tables, checking all six dimensions; failures create tickets in the data quality tracker. Layer 3 — **proactive anomaly detection**: ML-based monitoring (whylogs, Evidently AI) tracks distributions over time and alerts on drift before it causes visible problems. Every check has an owner, severity, and automated escalation path. + +**Q: What's the difference between data quality validation and data governance?** +A: Data quality validation is a specific set of technical practices (running checks, measuring dimensions, alerting on failures) within the broader data governance framework. Governance is the overall management structure — it defines who is responsible for quality (stewards), what the quality standards are (policies), and how quality issues are resolved (processes). Quality validation provides the data; governance provides the decision-making and accountability structure to act on that data. + +--- + +## Data Versioning + +**Definition:** Tracking and managing changes to datasets over time, analogous to code version control, enabling reproducibility and rollback. + +### Theory & Explanation + +Data versioning addresses a critical need: **reproducibility**. A machine learning model trained on data from two months ago should be reproducible with exactly the same data. Without versioning, source data changes (schema evolution, corrections, late-arriving records) make this impossible. **Auditability** requires knowing exactly what data was used for a specific report or model — regulators and stakeholders need this proof. **Collaboration** is improved when teams share consistent data snapshots rather than each member extracting their own copy. **Rollback** capability means a bad transformation can be undone by restoring the previous version. + +Three major approaches exist. **(1) DVC (Data Version Control)** — git-like versioning for data files stored in cloud storage (S3, GCS). DVC stores metadata (pointers + checksums) in git and the actual data in remote storage, enabling `git checkout` for code + `dvc checkout` for data together. **(2) Delta Lake time travel** — built-in versioning: `SELECT * FROM table VERSION AS OF 123` or `SELECT * FROM table TIMESTAMP AS OF '2024-03-15'`. Every write creates a new version; old versions are retained until vacuumed. **(3) lakeFS** — git-like branches for data lakes: create branches for experimental transformations, merge validated changes to main, and rollback with revert. + +The key challenge is **storage cost**. Every dataset change can create a new full copy. DVC uses diff-based versioning (only changed files are stored). Delta Lake stores only changed Parquet files per version. lakeFS uses copy-on-write to minimize storage overhead. Without these approaches, naive snapshotting (full COPY per version) becomes prohibitively expensive. + +### Example + +An ML team trains a churn prediction model. They use DVC: `dvc run -n train -d data/processed_features.csv -m metrics.json python train.py`. Every run versions the input data and output metrics. Three months later, they discover the latest feature engineering pipeline produced worse results than the previous version. They run `git log` to find the earlier commit, then `git checkout ` and `dvc checkout data/processed_features.csv`. The exact training data from three months ago is restored. They retrain the model and reproduce the earlier metrics exactly — proving the feature pipeline change caused the degradation. + +### Interview Questions + +**Q: How is data versioning different from data backups?** +A: Backups are full or incremental snapshots designed for disaster recovery — you restore the latest backup to recover from data loss. Versioning maintains a history of intentional changes over time for reproducibility, audit, and rollback. Backups are typically overwritten on a rotation schedule (daily backup kept for 30 days). Versioning keeps immutable snapshots keyed by timestamp or commit ID. A backup can be restored forward (latest state), but versioning allows you to travel backward or forward to any specific point in time. + +**Q: What's the storage cost trade-off of data versioning?** +A: Full versioning (complete copy per version) costs N × storage for N versions — expensive for large datasets. Modern approaches reduce this: DVC stores only changed files (diff-based); Delta Lake stores only new Parquet files per version (unchanged files are shared via Parquet's immutable design); lakeFS uses copy-on-write (only changed objects are new, unchanged objects are pointers to existing data). The trade-off is storage savings vs. complexity. For most ML use cases, retaining 30-90 days of granular versions with monthly long-term snapshots provides good coverage at reasonable cost. + +--- + +## Feature Store + +**Definition:** Centralized repository for storing, managing, and serving machine learning features consistently across training and inference. + +### Theory & Explanation + +The fundamental problem a feature store solves is **training-serving skew**: features computed during model training (batch, historical, offline) differ from features computed during inference (real-time, online) because the computation is implemented differently in each code path. A feature store provides a **single source of truth** for feature definitions — the same `average_ride_distance_7d` function is used for both training and serving. + +A feature store has two layers. The **offline store** stores large historical feature datasets for training — typically Parquet files in S3 or tables in BigQuery, supporting point-in-time queries for **time travel** joins (joining labels to features as they existed at prediction time, avoiding data leakage). The **online store** serves current feature values at low latency (< 10ms) for real-time inference — typically Redis, DynamoDB, or Cassandra. Feature computation can be **batch** (daily aggregations, windowed statistics) or **streaming** (real-time counts, rolling averages via Kafka/Flink). + +Key capabilities: **(1) Point-in-time correctness** — when training, each label is joined to feature values as they were at prediction time, not as they are today (which would include future information and cause leakage). **(2) Feature sharing** — teams discover and reuse features built by other teams, reducing redundant computation. **(3) Consistent serving** — training features and serving features are computed with identical logic. **(4) Feature monitoring** — track feature distributions over time to detect drift. + +### Example + +A ride-sharing company defines `average_ride_distance_7d` as a feature: the average ride distance for a driver over the past 7 days. During **training**, the feature store accepts a DataFrame of (driver_id, prediction_timestamp) pairs and returns the historical average for each timestamp — `WHERE ride_timestamp < prediction_timestamp AND ride_timestamp >= prediction_timestamp - 7 DAYS`. This guarantees no future information leaks into training. During **inference**, when a ride request comes in, the feature store returns the current running average from Redis in under 10ms. The same feature definition serves both paths. Two teams (ETA prediction and dynamic pricing) both use this feature from the shared store with identical semantics. + +### Interview Questions + +**Q: What is training-serving skew and how does a feature store prevent it?** +A: Training-serving skew occurs when feature computation differs between training and inference. For example, training code computes `avg_transaction_amount` over a 30-day window using SQL in Spark, but inference code computes it using a Python rolling average over the last 30 API events. Differences in window definition, null handling, data availability, or aggregation logic cause the model to see different values than it was trained on, degrading accuracy. A feature store prevents this by defining feature transformations once and executing them consistently in both the offline store (for training) and online store (for serving) using identical logic. + +**Q: Online store vs offline store — why both?** +A: The offline store is optimized for **throughput** — scanning billions of historical rows to create training datasets. It uses columnar storage (Parquet) and distributed query engines (Spark, BigQuery). Latency is seconds to minutes. The online store is optimized for **latency** — serving individual feature lookups at sub-10ms for real-time inference. It uses key-value storage (Redis, DynamoDB) and handles millions of requests per second. Both are needed because the same features must support two very different access patterns. + +--- + +## A/B Testing + +**Definition:** Controlled experiment comparing two variants (A = control, B = treatment) by randomly assigning users to groups and measuring the difference in a target metric. + +### Theory & Explanation + +A/B testing is the gold standard for causal inference in product and business decisions. Core principles: **Randomization** ensures each user has an equal chance of assignment, creating statistically equivalent groups so that any observed difference can be attributed to the treatment. **Sample size determination** uses power analysis: significance level α = 0.05 (5% chance of Type I error — false positive), power β = 0.80 (20% chance of Type II error — false negative), and the **Minimum Detectable Effect (MDE)** — the smallest effect size worth detecting. Sample size increases as MDE decreases: detecting a 0.1% lift requires much more data than a 5% lift. + +The **peeking problem** occurs when experimenters check results repeatedly during data collection and stop early when results reach significance. This inflates the Type I error rate dramatically — checking daily can push false positive rates to 20-30% or higher. Solutions: (1) pre-register a fixed sample size and never peek; (2) use sequential testing methods (always valid p-values, group sequential designs) that allow monitoring without inflating error rates. + +**Metrics** are categorized as: **Primary metric** — the target of optimization (conversion rate, revenue per user). **Secondary metrics** — guardrail metrics that should not degrade (page load time, customer support tickets). **Proxy metrics** — correlated with the true metric but measurable earlier/faster. **Statistical tests**: chi-square for proportion-based metrics (conversion rate), t-test for continuous metrics (revenue per user), Mann-Whitney for non-normal distributions. **Simpson's paradox** can mask or reverse effects — always disaggregate results across segments (device type, country, traffic source). + +**Practical significance** matters more than statistical significance. A statistically significant result (p < 0.05) with a tiny effect size (0.01% lift on conversion) may not be worth implementing. Compute the expected business impact: +0.3 percentage points × 10M annual transactions × $50 average order = $1.5M. That is practically significant. + +### Example + +An e-commerce company tests a new checkout button color. 50,000 users are randomly assigned to control (blue button) and 50,000 to treatment (green button). Control conversion rate = 3.2% (1,600 purchases); treatment conversion rate = 3.5% (1,750 purchases). A chi-square test yields: χ² = 6.84, p = 0.009. Since p < 0.05, the result is statistically significant. The relative lift = (3.5% / 3.2% - 1) × 100% = +9.4%. Practical impact: +0.3 percentage points × 5M annual purchasers × $80 average order = $1.2M/year. Guardrail metrics (page load time, error rate) are unchanged. The green button is rolled out to all users. + +### Interview Questions + +**Q: How many users do you need for an A/B test?** +A: The required sample size depends on α (typically 0.05), β (typically 0.20), and MDE. Formula: n = (Z_α/2 + Z_β)² × (p₁(1-p₁) + p₂(1-p₂)) / (p₂ - p₁)² for proportions. For a conversion rate improving from 3.2% to 3.5% (MDE = 0.3pp), n ≈ 50,000 per group. For detecting a 0.1pp lift from 3.2%, n ≈ 450,000 per group. A power analysis calculator should be used before every experiment to ensure the test is adequately powered. + +**Q: What's the peeking problem and how do you avoid it?** +A: Peeking means checking results early and potentially stopping based on data-dependent significance. Each peek inflates the Type I error rate — after 50 peeks, a truly null result has ~40% chance of being declared significant. Avoid it by: (1) pre-registering sample size and not looking at results until the data collection is complete; (2) using sequential testing methods (Lan-DeMets spending function, always-valid p-values) that correct for multiple looks; (3) implementing automated stopping rules in the experimentation platform rather than manual checking. + +**Q: How would you analyze results if the test was not properly randomized?** +A: If randomization is violated (e.g., treatment assigned by userID parity, geography, or self-selection), groups may differ on confounding variables. Use causal inference methods: (1) propensity score matching — estimate the probability of treatment assignment and match treated/control units on propensity scores; (2) difference-in-differences — compare the change in treatment group against the change in control group over time; (3) instrumental variables — if a randomized instrument exists; (4) CUPED (Controlled-experiment using Pre-Experiment Data) — use pre-experiment covariates to reduce bias. Document the violation transparently and caveat conclusions accordingly. + +--- + +## Semi-supervised Learning + +**Definition:** Combines small labeled data with large unlabeled data, leveraging the structure of the data distribution to improve performance when labels are scarce. + +### Theory & Explanation + +Semi-supervised learning (SSL) is based on three core assumptions. **Smoothness assumption**: if two points are close in the input space, their labels should be similar. **Cluster assumption**: data forms clusters, and points in the same cluster share the same label — decision boundaries should lie in low-density regions. **Manifold assumption**: high-dimensional data lies on a lower-dimensional manifold, and learning can be done on that manifold. + +Self-training is the simplest approach: train a model on labeled data, use it to predict pseudo-labels on unlabeled data, select high-confidence predictions, add them to the training set, and retrain. The risk is **confirmation bias** — once the model starts making incorrect high-confidence predictions, errors reinforce themselves. **Co-training** splits features into two independent views, trains separate models on each, and uses the highest-confidence predictions from one view to augment the other. + +**Consistency regularization** (FixMatch, MixMatch) is the current state of the art. It applies weak augmentation (standard flip-and-crop) to unlabeled data to generate pseudo-labels, then applies strong augmentation (RandAugment, CTA) to the same data and trains the model to predict the same label under heavy augmentation — enforcing augmentation invariance. FixMatch combines this with a confidence threshold: only pseudo-labels with probability above a threshold (e.g., 0.95) are used, dramatically reducing confirmation bias. + +### Example + +A medical imaging dataset has 500 labeled X-ray images and 50,000 unlabeled images. A FixMatch model is trained: labeled images train with standard cross-entropy; unlabeled images are weakly augmented to generate pseudo-labels with confidence threshold 0.95, then strongly augmented to train the consistency loss. The model achieves 93% accuracy on the test set — compared to 82% with only the 500 labeled samples (standard supervised learning). The key is the SSL assumption that bone structure variations in X-rays lie on a low-dimensional manifold that unlabeled data helps map. + +### Interview Questions + +**Q: What assumptions must hold for SSL to work?** +A: SSL works when the data distribution has structure that unlabeled data can help reveal. The cluster assumption must hold — decision boundaries should fall in low-density regions. The smoothness assumption must hold — nearby points should have similar labels. The manifold assumption must hold — the data must lie on a lower-dimensional manifold. If these assumptions are violated (e.g., classes are highly overlapping, or the data is pure noise), SSL can actually degrade performance. + +**Q: Confirmation bias in SSL — how to address?** +A: Confirmation bias arises when the model's own predictions reinforce errors — incorrect high-confidence pseudo-labels from early training persist and contaminate the model. Mitigation strategies: (1) use a confidence threshold (only pseudo-labels above 0.95 are used); (2) use strong augmentation on unlabeled data so predictions are tested against transformation invariance; (3) employ a diverse ensemble of models for pseudo-label generation; (4) anneal the weight of the unsupervised loss, starting low and increasing as the model improves; (5) use MixMatch-like approaches that sharpen predictions without hard thresholding. + +--- + +## Reinforcement Learning + +**Definition:** An agent learns sequential decisions through trial-and-error interaction with an environment, maximizing cumulative reward signals. + +### Theory & Explanation + +Reinforcement Learning (RL) is formalized as a **Markov Decision Process (MDP)**: a tuple (S, A, P, R, γ) where S is the set of states, A is the set of actions, P(s' | s, a) is the transition probability, R(s, a) is the immediate reward, and γ (discount factor) determines how much future rewards are valued. The agent learns a **policy** π(a | s) — a mapping from states to actions — that maximizes expected cumulative discounted reward. + +The **exploration vs exploitation** dilemma is central: the agent must try new actions (explore) to discover better strategies while also exploiting known good actions for reward. Common strategies: ε-greedy (take random action with probability ε, decaying over time), Upper Confidence Bound (UCB — choose actions with high uncertainty), and Thompson sampling (sample from posterior distribution of action values). + +Algorithm families: **Value-based** methods estimate the optimal value function Q*(s, a) — the expected return from taking action a in state s. Q-learning learns Q(s, a) via the Bellman update: Q(s, a) ← Q(s, a) + α[R + γ max_a' Q(s', a') - Q(s, a)]. DQN extends this with deep neural networks and experience replay. **Policy-based** methods (REINFORCE, PPO) directly optimize the policy π(a | s) using gradient ascent on expected reward. **Actor-Critic** methods (A3C, SAC) combine both — the actor learns the policy, the critic learns the value function. Applications include game-playing (AlphaGo, Dota 2), robotics, autonomous driving, and RLHF (reinforcement learning from human feedback) for fine-tuning LLMs. + +### Example + +An agent learns to navigate a 5×5 grid world. The goal position (+10 reward) is in the top-right corner; pits (−5 reward) are in two other cells. Q-learning initializes Q(s, a) = 0 for all state-action pairs. The agent explores with ε = 1.0, decaying to 0.1 over 1,000 episodes. Initially, it takes random actions, occasionally reaching the goal and updating Q-values. Over time, the optimal path emerges: the Q-values propagate backward from the goal via the Bellman equation. After convergence, the agent follows the optimal path every episode, avoiding pits. The learned policy generalizes — even if starting position changes, the agent navigates optimally from the new state. + +### Interview Questions + +**Q: On-policy vs off-policy — what's the difference?** +A: On-policy methods evaluate or improve the same policy that generates behavior. The agent must learn from experience generated by the current policy — changing the policy invalidates previous experience. SARSA is on-policy. Off-policy methods can learn from experience generated by any behavior policy. Q-learning is off-policy: it learns the optimal Q-function regardless of how actions were chosen. This makes Q-learning more sample-efficient (it can reuse past experience) and compatible with experience replay. The trade-off: off-policy methods have higher variance and can be less stable. + +**Q: Credit assignment problem in RL?** +A: The credit assignment problem is determining which actions in a sequence were responsible for a delayed reward. A robot might take 100 actions before reaching a goal — which actions were critical? The naive approach (reward each action equally) is incorrect. Temporal difference learning addresses this by bootstrapping: current Q-value is updated toward the current reward plus the estimated value of the next state, propagating credit one step at a time. Eligibility traces (λ-returns) extend this by mixing multi-step returns, providing a smooth trade-off between bias and variance in credit assignment. + +**Q: When to use PPO vs DQN?** +A: PPO (Proximal Policy Optimization) is preferred when: (1) the action space is continuous (robotics, autonomous driving); (2) the action space is high-dimensional; (3) you need stable, monotonic improvement; (4) the environment is partially observable. DQN is preferred when: (1) the action space is discrete and small; (2) experience replay is beneficial; (3) the environment is deterministic or near-deterministic. For most modern RL problems, PPO or SAC (Soft Actor-Critic) are the default choices unless the action space is small and discrete. + +--- + +## Bias (Data) + +**Definition:** Systematic errors in data collection, sampling, or labeling that cause the data to misrepresent the true distribution of the phenomenon being studied. + +### Theory & Explanation + +Data bias takes many forms. **Sampling bias** occurs when the sampling process does not reflect the population — surveying only smartphone users about internet access misses the offline population. **Measurement bias** occurs when data collection instruments systematically error — a poorly calibrated sensor reads temperature 2°C low, or a survey question is leading. **Reporting bias** occurs when certain observations are more likely to be recorded — people with severe side effects are more likely to report them than those with mild symptoms. **Survivorship bias** occurs when only surviving or successful cases are observed — analyzing successful startups without considering the thousands that failed. **Confirmation bias** occurs when data collection favors pre-existing beliefs — labeling errors that align with annotator expectations. + +Bias propagates from data through models. A model trained on biased data learns and amplifies the bias. **Detection** requires: statistical audits comparing dataset distributions to known population distributions; **disaggregated evaluation** — measuring model performance separately across demographic groups, income levels, geographic regions; and **fairness metrics** — demographic parity (equal prediction rates across groups), equal opportunity (equal true positive rates), equalized odds (equal TPR and FPR). + +**Mitigation** happens at three stages. **Pre-processing**: reweight training samples to match population proportions; resample to balance groups; transform features to remove protected attributes. **In-processing**: add fairness constraints to the loss function; use adversarial debiasing (train to maximize accuracy while minimizing an adversary's ability to predict the protected attribute). **Post-processing**: adjust decision thresholds independently per group to achieve parity; calibrate predictions after model output. + +### Example + +A hospital trains a readmission prediction model on five years of data (2018-2022). Due to COVID, 2020 admissions were disproportionately elderly patients. The model learns that "age > 65" is a strong predictor, but the 2020 oversampling of elderly patients (who have different readmission patterns) biases this relationship. In production on 2024 data, the model over-predicts readmission risk for all elderly patients. Detection: the Population Stability Index (PSI) comparing training age distribution to deployment distribution flags significant drift. Mitigation: reweight 2020 samples to match the expected population age distribution, reducing their influence. + +### Interview Questions + +**Q: How to detect bias in production ML?** +A: Implement continuous monitoring at three levels: (1) **Input monitoring** — track feature distribution shifts (PSI, KS statistic) between training and production; flag demographic representation shifts. (2) **Model monitoring** — track prediction distribution changes (is the model predicting positive outcomes at different rates across groups?). (3) **Outcome monitoring** — the most important but hardest — track actual outcomes when available. Compare false positive rates and false negative rates across demographic segments using disaggregated dashboards. When disparities exceed thresholds, trigger investigation and model retraining. + +**Q: Survivorship bias — give an ML example.** +A: A model trained to predict startup success uses a dataset of successfully funded startups from CrunchBase. It learns that "raised Series A" is a strong feature. This is survivorship bias — the dataset only contains startups that survived long enough to raise funding. The model cannot learn the patterns of failure (which are crucial to predicting success). In practice, the model would recommend investing in startups that look like past successes but miss warning signs that only appear in failed startups' data. Countermeasure: deliberately include failed startups (from acquisition databases, bankruptcy filings) and label them as negative cases. + +**Q: Label bias vs measurement bias?** +A: Label bias (annotation bias) occurs when the ground-truth labels themselves are systematically wrong — annotators favor certain outcomes, or the labeling guidelines encode subjective judgments. For example, labeling toxicity in comments where annotators disagree about sarcasm. Measurement bias occurs when the input features are systematically distorted — survey responses where socially undesirable answers are under-reported. Label bias corrupts the target variable; measurement bias corrupts the features. Both harm model quality but require different fixes: label bias needs better annotation guidelines; measurement bias needs better sensors or survey design. + +--- + +## Imbalanced Data + +**Definition:** Classification datasets with heavily skewed class distributions where the minority class is typically the class of interest. + +### Theory & Explanation + +Standard machine learning models optimize for overall accuracy, which is misleading on imbalanced data. A fraud detector that predicts "not fraud" for every transaction achieves 99.8% accuracy — but fails entirely at its purpose. This is the **accuracy paradox**: high accuracy despite zero predictive value for the minority class. + +**Resampling** balances classes. **Random undersampling** removes majority class samples randomly — simple but loses potentially useful data. **SMOTE (Synthetic Minority Oversampling Technique)** generates synthetic minority samples by interpolating between existing minority points: x_new = x_i + λ(x_j - x_i) where x_i and x_j are k-nearest neighbors from the minority class and λ ~ Uniform(0, 1). SMOTE creates plausible new examples rather than duplicating existing ones. **ADASYN** adaptively generates more synthetic samples in regions where the minority class is hardest to learn. + +**Algorithmic approaches** include: **Class weighting** — assign higher misclassification cost to minority class errors, typically inversely proportional to class frequency (weight = N_majority / N_minority). **Cost-sensitive learning** — incorporate a cost matrix into the loss function. **Focal loss** — down-weights well-classified examples, forcing the model to focus on hard minority examples. + +**Evaluation** must use metrics not distorted by imbalance: **Precision** (TP / (TP + FP)), **Recall** (TP / (TP + FN)), **F1-score** (harmonic mean of precision and recall), **PR-AUC** (precision-recall area under curve — much better than ROC-AUC for imbalance), and **MCC** (Matthews correlation coefficient — balanced measure even with severe skew). + +### Example + +Credit card fraud detection: the dataset has 492 fraudulent transactions out of 284,807 total (0.17%). A baseline model predicting "not fraud" for everything achieves 99.83% accuracy but 0% recall. After applying SMOTE: synthetic fraud samples are generated to create a 50/50 balanced training set. The model now achieves 92% recall (up from a naive model's 30%) at 85% precision. Alternatively, class weighting assigns weight 1000× to fraud samples in the loss function, achieving 88% recall with less synthetic data overhead. PR-AUC improves from 0.15 (baseline) to 0.72 (SMOTE) to 0.68 (class weighting). + +### Interview Questions + +**Q: Why is ROC-AUC misleading for imbalanced data?** +A: ROC-AUC plots TPR (true positive rate) against FPR (false positive rate). With high imbalance, FPR is dominated by the huge number of true negatives — even a large number of false positives barely moves FPR. The curve can look excellent while the model performs poorly on the minority class. For example, with 99.8% majority class, FPR = 0.01 means only 0.01 × 284,000 ≈ 2,840 false positives, but that may be unacceptable for fraud detection. **PR-AUC** is better because precision directly accounts for false positives relative to the minority class size: precision = TP / (TP + FP) — if the minority class is tiny, any FP significantly hurts precision. + +**Q: SMOTE vs class weighting — when to use each?** +A: SMOTE is preferred when: (1) the minority class has very few samples (< 100) and the model cannot learn the decision boundary without more data; (2) the minority class is well-clustered so interpolation produces realistic samples; (3) the dataset is not excessively large (SMOTE increases dataset size). Class weighting is preferred when: (1) the minority class has enough samples to define a decision boundary (hundreds+); (2) the dataset is large enough that SMOTE would be computationally expensive; (3) you want to avoid any risk of generating unrealistic synthetic data; (4) you're using a probabilistic model that responds well to cost weighting (logistic regression, neural networks). In practice, many practitioners try both and select by validation performance. + +**Q: Multi-class imbalance strategies?** +A: Extend binary approaches: (1) **One-vs-Rest** — treat each class as binary (minority vs rest), apply SMOTE or weighting per class. (2) **Hierarchical** — group rare classes into a superclass, classify fine-grained only when confident. (3) **Focal loss** extends naturally to multi-class with class-specific focusing parameters. (4) **Class-balanced loss** — weight each class by 1 / (1 - β^n_c) where n_c is class sample count and β is a smoothing parameter. (5) **Decoupled training** — first learn representations on imbalanced data, then fine-tune the classifier layer with balanced sampling. + +--- + +## EDA (Exploratory Data Analysis) + +**Definition:** An investigative approach to understanding data distributions, patterns, anomalies, and relationships before formal modeling or hypothesis testing. + +### Theory & Explanation + +EDA follows a structured progression from simple to complex. **Univariate analysis** examines each variable in isolation: histograms for distributions, box plots for outliers and quartiles, summary statistics (mean, median, standard deviation, skewness, kurtosis), and Q-Q plots for normality assessment. **Bivariate analysis** explores relationships between pairs of variables: scatter plots for continuous-continuous relationships, grouped box plots for continuous-categorical, stacked bar charts for categorical-categorical, and correlation matrices (Pearson for linear, Spearman for monotonic). **Multivariate analysis** explores interactions among multiple variables: pair plots (scatter matrix), PCA/t-SNE/UMAP for dimensionality reduction and cluster visualization, parallel coordinates, and heatmaps of interaction effects. + +The purpose is exploration, not confirmation. EDA uncovers patterns that generate hypotheses — it does not test them. Specific goals include understanding **distributions** (are they normal, skewed, bimodal, multimodal?), detecting **anomalies** (extreme values, impossible combinations, missing data patterns), testing **assumptions** (linearity, homoscedasticity, normality for downstream tests), and generating **hypotheses** for formal testing. + +EDA typically consumes 60-80% of project time. Tools: pandas-profiling (now ydata-profiling) for automated univariate reports; matplotlib and seaborn for static visualizations; plotly for interactive exploration; missingno for visualizing missing data patterns. The best EDA combines automated profiling with targeted manual investigation guided by domain knowledge. + +### Example + +EDA on a loan application dataset reveals: (1) credit score is bimodal with peaks at 620 and 750 — indicating two distinct populations; (2) 50 records show income < $20,000 with loan amounts > $100,000 — identified as suspicious entries for further investigation; (3) employment length has a **missing not at random (MNAR)** pattern — missing values are concentrated in the low credit score segment (likely unemployed or self-employed applicants who skip the field); (4) young applicants (< 25) with high loan amounts (> 3× income) have a 40% default rate compared to 5% overall — a strong risk segment identified during multivariate analysis. + +### Interview Questions + +**Q: Walk through your EDA process on a new dataset.** +A: (1) Load and inspect: check shape, data types, first/last rows, summary statistics. (2) Missing data analysis: percentage of missing values per column, visualize missing patterns (missingno), determine MCAR/MAR/MNAR. (3) Univariate: histograms + box plots for numeric columns, value counts + bar charts for categorical columns, identify outliers and distribution shapes. (4) Bivariate: correlation heatmap (limit to numeric), cross-tabulations for categorical pairs, scatter plots for promising numerics. (5) Targeted: follow up on interesting patterns — why is this feature bimodal? Why are these records outliers? (6) Summary: document findings, generate hypotheses, note data quality issues, produce visualization dashboard for stakeholder review. + +**Q: High-dimensional EDA (1000+ features) — how do you approach it?** +A: Individual feature plots are impractical at 1000+ dimensions. Strategy: (1) Start with automated profiling (pandas-profiling) to flag distributions, missing rates, and outliers. (2) Use PCA / t-SNE / UMAP to visualize global structure — do natural clusters emerge? (3) Filter by variance — remove near-zero variance features. (4) Correlation clustering — group highly correlated features, select representative from each cluster. (5) Target-based filtering — compute univariate association with target (mutual information, AUC) for each feature; examine top 30-50 features individually. (6) Regularized models (Lasso, Ridge) for feature selection — which features survive regularization? (7) SHAP or permutation importance on an initial model to understand feature contributions. + +**Q: EDA vs hypothesis testing — when does exploration end?** +A: EDA is for discovery and hypothesis generation; hypothesis testing is for confirmation. The transition happens once you have a specific, testable claim — "customers under 25 with high loan amounts have higher default rates" becomes a hypothesis to be tested on a holdout set or with a formal statistical test (chi-square or logistic regression). The danger of mixing them is **data dredging** (p-hacking) — if you explore until you find a pattern and then declare it statistically significant without adjusting for multiple comparisons, your conclusions are unreliable. Best practice: set aside a holdout dataset before EDA begins, and only use it for confirmatory testing of hypotheses discovered during exploration. + +--- + +## Shapiro-Wilk Test + +**Definition:** A statistical test for normality with the null hypothesis H₀ that the sample is drawn from a normal distribution. The W statistic measures how well ordered sample values match expected normal order statistics. + +### Theory & Explanation + +The Shapiro-Wilk test computes the W statistic: W = (Σ a_i x_(i))² / Σ (x_i - x̄)², where x_(i) are the ordered sample values and a_i are constants derived from the expected values of normal order statistics. The numerator captures how well the ordered sample correlates with the expected values under normality. W ranges from 0 to 1 — values close to 1 indicate normality; small values indicate departure from normality. + +Shapiro-Wilk is considered the **most powerful** normality test for general alternatives — it has better statistical power (ability to detect non-normality) than Kolmogorov-Smirnov, Anderson-Darling, or the D'Agostino-Pearson test for most distributions. Implementation is straightforward: `scipy.stats.shapiro(x)` returns the W statistic and p-value. If p < α (typically 0.05), reject H₀ and conclude the data is not normal. + +Sample size sensitivity is critical. For **n < 20**, the test has **low power** — it may fail to detect even strong non-normality. For **n > 5,000**, the test detects **trivial deviations** — even data from a nearly-normal distribution will reject H₀ because real-world data is never perfectly normal. The recommended range is n = 20 to n = 2,000. For larger samples, rely on Q-Q plots and skewness/kurtosis values rather than the p-value alone. + +### Example + +A researcher measures 30 reaction times (in milliseconds): [245, 312, 298, 421, 287, 305, ...]. Shapiro-Wilk returns W = 0.924, p = 0.037. Since p < 0.05, the null hypothesis of normality is rejected. The researcher examines a histogram — the distribution is right-skewed with several slow responses. After log transformation, Shapiro-Wilk on log(rt): W = 0.971, p = 0.58. The transformed data is consistent with normality. Decision: use the Mann-Whitney U test (non-parametric) for group comparisons on original data, or use log-transformed data with a t-test. + +### Interview Questions + +**Q: Why does Shapiro-Wilk fail on large samples?** +A: Real-world data is never perfectly normal — there will always be minor deviations (slight skew, minor kurtosis, rounding). With large n, the test has sufficient power to detect these trivial deviations as statistically significant. A sample of 10,000 with skewness 0.1 and kurtosis 3.1 (very nearly normal) will likely produce p < 0.001, rejecting normality. The correct interpretation is not "the data is severely non-normal" but "the sample is large enough to detect a negligible departure." For large samples, assess normality visually (Q-Q plot) and by effect size (skewness < |1|, kurtosis < |3|) rather than by p-value. + +**Q: Shapiro-Wilk vs Kolmogorov-Smirnov — which should you use?** +A: Shapiro-Wilk is generally preferred for normality testing because it is more powerful — it detects non-normality with smaller sample sizes. The Kolmogorov-Smirnov test (Lilliefors-corrected for estimated parameters) compares the empirical distribution function to the normal CDF but has weaker power, especially for detecting tail deviations. Standard practice: use Shapiro-Wilk for n = 20-2,000; use Q-Q plots + skewness/kurtosis for larger samples; avoid KS for normality testing unless you specifically need a distribution-free test. + +**Q: Does p > 0.05 prove normality?** +A: No. p > 0.05 means the test did not detect a significant departure from normality — it does not prove the data is normal. The failure to reject H₀ could be due to genuinely normal data, but equally could be due to insufficient power (small sample size) or the test being insensitive to the specific type of non-normality present. Always assess normality using multiple methods: Shapiro-Wilk, Q-Q plots, skewness (< |1|), kurtosis (< |3|), and histograms. Consistency across methods provides confidence. + +--- + +## Levene's Test + +**Definition:** A statistical test for homogeneity of variances across groups, with the null hypothesis H₀ that group variances are equal. Used as a prerequisite for ANOVA and two-sample t-tests. + +### Theory & Explanation + +Levene's test assesses whether k groups have equal variances — a key assumption of standard ANOVA and the two-sample t-test (which assume homoscedasticity). The test computes absolute deviations of each observation from its group's central tendency, then performs an ANOVA on those deviations. If the group mean absolute deviations differ significantly, the null hypothesis of equal variances is rejected. + +The test has three variants depending on the central tendency used. **Mean-based** (using group means): most powerful under normality but sensitive to non-normality. **Median-based** (default in most implementations): more robust to non-normality and is the recommended default. **Trimmed mean-based** (10% trimmed mean): a compromise — retains some power under normality while improving robustness. The median version is also known as the **Brown-Forsythe test**. Implementation: `scipy.stats.levene(*groups, center='median')`. + +Levene's test is **less sensitive to non-normality** than alternatives: **Bartlett's test** is the most powerful when data is perfectly normal but extremely sensitive to non-normality (a small departure inflates Type I error). The **F-test of variances** (ratio of two variances) is similarly non-robust and should only be used for normal data. Levene's is the safe default for most applications. + +### Example + +A researcher compares three teaching methods. Group A variance = 45, Group B variance = 120, Group C variance = 50. Levene's test with median center: W = 4.28, p = 0.018. Since p < 0.05, the null hypothesis of equal variances is rejected. The standard ANOVA cannot be used because it assumes homoscedasticity. Instead, the researcher uses **Welch's ANOVA** (which does not assume equal variances) or transforms the outcome variable (log transform reduces variance differences: after log transform, Levene's W = 1.89, p = 0.16 — variances are now homogeneous). + +### Interview Questions + +**Q: Why not always use Welch's t-test instead of testing for equal variances?** +A: This is a reasonable position — many statisticians recommend always using Welch's t-test as the default, bypassing Levene's test entirely. Welch's t-test does not assume equal variances and performs nearly as well as the standard t-test when variances are equal. The argument for checking first: (1) in ANOVA with 3+ groups, Welch's ANOVA is not always supported in all software; (2) if variances are equal, the standard test has slightly more power; (3) knowing about variance heterogeneity is itself informative about the data (treatment may affect variance as well as mean). In practice, for two-group comparisons, default to Welch's; for multi-group ANOVA, check with Levene's. + +**Q: Levene's vs Bartlett's test — which is better?** +A: Levene's is better for general use because it is robust to non-normality — it produces valid Type I error rates even when the data is not normal. Bartlett's test is more powerful than Levene's *when the data is exactly normal*, but extremely sensitive to non-normality (a small departure inflates false positives dramatically). Since real data is rarely perfectly normal, Levene's is the safer default. Use Bartlett's only when you know the data is normally distributed (checked via Shapiro-Wilk or Q-Q plots). + +**Q: If Levene's test is significant, can you still run ANOVA?** +A: You should not run standard ANOVA if Levene's test rejects equal variances, because heteroscedasticity inflates the Type I error rate (you are more likely to find a false significant result). Alternatives: (1) **Welch's ANOVA** — does not assume equal variances, available in most statistical software; (2) **Kruskal-Wallis test** — non-parametric alternative that does not assume normality or equal variances; (3) **transform the outcome** — log, square root, or Box-Cox transformation may stabilize variances; (4) **robust regression** — uses heteroscedasticity-consistent standard errors. Always report which approach was used and why. + +--- + +## Data Science Workflow + +**Definition:** A structured 10-stage process from business problem framing to deployed, monitored solution that guides data science projects from conception to production. + +### Theory & Explanation + +The data science workflow provides a systematic framework for tackling data problems. Each stage has specific objectives and deliverables, and the process is **iterative** — insights at later stages often force revisiting earlier ones. + +**Stage 1 — Business Problem Framing:** Translate a business need into a well-defined data problem. Ask: What exactly are we trying to optimize? What is the current baseline? What would success look like? What decisions will be made based on the output? Define metrics (business metrics and ML metrics). Deliverable: project charter with problem statement, success criteria, and stakeholder alignment. + +**Stage 2 — Data Collection:** Identify and acquire necessary data sources. Includes internal databases, third-party APIs, web scraping, sensor data, or manual collection. Assess data availability, access permissions, and cost. Deliverable: data source inventory with schema documentation. + +**Stage 3 — EDA:** Explore data distributions, relationships, missing patterns, anomalies, and quality issues. Generate hypotheses. Deliverable: EDA report with visualizations and initial findings. + +**Stage 4 — Data Cleaning:** Handle missing values (imputation, deletion, flagging), fix data types, remove or cap outliers, resolve inconsistencies, deduplicate. Deliverable: clean, validated dataset. + +**Stage 5 — Feature Engineering:** Create predictive features from raw data — transformations, aggregations, encodings, interactions, domain-specific features. Select features using domain knowledge and statistical methods. Deliverable: feature set ready for modeling. + +**Stage 6 — Model Selection:** Train multiple candidate models (baseline → simple → complex). Use cross-validation for robust performance estimates. Compare on validation metrics. Deliverable: shortlist of candidate models with performance profiles. + +**Stage 7 — Hyperparameter Tuning:** Optimize model hyperparameters using grid search, random search, or Bayesian optimization. Validate tuned model on held-out validation set (or nested CV). Deliverable: final tuned model. + +**Stage 8 — Evaluation:** Final evaluation on held-out test set. Assess not just overall metrics but disaggregated performance across segments. Check for calibration, fairness, and robustness. Deliverable: evaluation report with model card. + +**Stage 9 — Deployment:** Package model (Docker, MLflow), deploy as API or batch inference, integrate with application, set up monitoring infrastructure. Deliverable: deployed model serving predictions. + +**Stage 10 — Monitoring and Maintenance:** Track prediction distributions, feature distributions, data drift, concept drift, model performance over time. Set up automated retraining triggers and alerting for degradation. Deliverable: monitoring dashboard and retraining pipeline. + +### Example + +An e-commerce company tackles cart abandonment. **Stage 1**: "Predict which users will abandon their cart within 30 minutes so we can send a targeted discount." Success metric: cart abandonment rate reduction from 78% to 65%. **Stage 2**: Collect clickstream events, cart state changes, user profiles, historical purchase data. **Stage 3**: EDA reveals average time-to-abandon is 14 minutes, 40% of abandonment happens on mobile, price-sensitive users abandon most. **Stage 4**: Clean session timestamps, impute missing device types, remove bot traffic. **Stage 5**: Engineer features — session duration, pages viewed, cart value, discount history, time of day, device type, price sensitivity score. **Stage 6**: Compare logistic regression (baseline), XGBoost, and LightGBM. XGBoost wins with 0.85 PR-AUC. **Stage 7**: Tune max_depth, learning_rate, subsample via random search. **Stage 8**: Final evaluation: 0.87 PR-AUC on test set, 82% recall at 60% precision, performance consistent across mobile/desktop. **Stage 9**: Deploy as Flask API on Kubernetes with 100ms latency SLA. **Stage 10**: Monitor prediction distribution — after 3 months, drift in price sensitivity feature triggers retraining. Result: cart abandonment rate drops from 78% to 61%, exceeding the target. + +### Interview Questions + +**Q: What is the most common mistake in the data science workflow?** +A: **Jumping to modeling too early** — skipping or rushing Stages 1-4 (business framing, data collection, EDA, data cleaning). Teams often start with modeling because it is more interesting, but poor problem framing leads to building the wrong thing, and poor data quality makes any model unreliable. The sunk cost of a week of ETL and EDA is far less than two months building a model that doesn't address the right problem or fails in production. A related mistake is conflating correlation discovered in EDA with causation — exploration generates hypotheses, it does not confirm them. + +**Q: Model complexity vs interpretability — how do you balance them?** +A: The balance depends on the use case. For high-stakes decisions (medical diagnosis, credit approval, criminal justice), interpretability is essential — you must explain why a decision was made. Use simple models (logistic regression, decision trees, GAMs) or post-hoc interpretation methods (SHAP, LIME). For low-stakes but high-leverage tasks (recommendation, ad ranking, content moderation), complex models (gradient boosting, deep learning) are justified for their accuracy gains. The strategy: always start with a simple, interpretable baseline to establish a performance floor and understand the data. Only add complexity when it provides meaningful accuracy gains that outweigh the interpretability loss. + +**Q: How to detect concept drift in production?** +A: Concept drift means the relationship between features and target changes over time. Detection methods: (1) **monitor prediction distribution** — if the model's score distribution shifts significantly, it may indicate drift; (2) **monitor actuals** — when ground truth arrives with a delay, compare predicted vs actual via drifted performance metrics; (3) **domain classifier** — train a classifier to distinguish current production data from training data; high accuracy indicates drift; (4) **statistical tests** — compare feature distributions between a recent window and the training data using KS test or PSI. Automate monitoring with tools like Evidently AI, WhyLabs, or NannyML. When drift exceeds thresholds, trigger retraining or investigation. diff --git a/techy/02-machine-learning.md b/techy/02-machine-learning.md new file mode 100644 index 0000000..9f7991d --- /dev/null +++ b/techy/02-machine-learning.md @@ -0,0 +1,642 @@ +# 2. MACHINE LEARNING + +## Linear Regression +**Definition:** Fits line y=mx+b minimizing MSE via OLS solution. Simplest, most interpretable regression algorithm. + +### Theory & Explanation (3-5 paragraphs) + +Linear regression assumes the relationship between the target variable y and the feature matrix X is approximately linear. The model takes the form y = w₀ + w₁x₁ + w₂x₂ + ... + wₚxₚ + ε, where w are the coefficients (weights) and ε is the irreducible error term. The Ordinary Least Squares (OLS) solution finds the coefficients that minimize the sum of squared residuals: min_w ||Xw - y||₂². This has a beautiful closed-form solution: ŵ = (XᵀX)⁻¹Xᵀy, making linear regression one of the few ML algorithms with a direct analytical solution rather than requiring iterative optimization. + +For the model to produce reliable coefficient estimates and valid inference, several assumptions must hold. **Linearity** — the relationship between X and y must be linear (violation leads to biased estimates). **Independence** — observations must be independent (violated in time series data). **Homoscedasticity** — constant variance of residuals across all levels of X (heteroscedasticity inflates standard errors). **Normality of residuals** — needed for valid hypothesis tests and confidence intervals (less critical for large samples due to CLT). **No multicollinearity** — predictors should not be highly correlated (violation inflates coefficient variances and makes interpretation unreliable). In practice, real-world data rarely satisfies all assumptions perfectly, which is why diagnostic plots (residuals vs fitted, Q-Q plot, scale-location) are essential. + +Key evaluation metrics include R² (proportion of variance explained), Adjusted R² (penalizes unnecessary features), RMSE (root mean squared error in original units), and MAE (mean absolute error). When linear regression suffers from high variance or multicollinearity, regularized variants help: **Ridge (L2)** shrinks coefficients toward zero but never to exactly zero, useful when all features matter. **Lasso (L1)** performs feature selection by driving coefficients to exactly zero, ideal for sparse models. **ElasticNet** combines both L1 and L2 penalties, balancing feature selection with coefficient shrinkage. Regularization strength is controlled by λ (or alpha), tuned via cross-validation. + +### Example + +**House price prediction:** Model fitted on 2000 homes gives price = 50000 + 150 × sqft + 10000 × bedrooms. For a 1500 sqft, 3-bedroom house: predicted price = 50000 + 150(1500) + 10000(3) = 50000 + 225000 + 30000 = $305,000. R² = 0.76 meaning 76% of price variance is explained by these two features. RMSE = $42,000 (typical prediction error). Residual diagnostics show mild heteroscedasticity (larger errors for expensive homes) and a slight right-skew in residuals. Log-transforming price improves R² to 0.81 and stabilizes residual variance. + +### Interview Questions (2-3) + +**Q: What are the key assumptions of linear regression and what happens when they are violated?** +A: The five key assumptions are linearity, independence, homoscedasticity, normality of residuals, and no multicollinearity. Violating linearity leads to biased estimates — the model systematically mispredicts. Violating independence (e.g., time series data) underestimates standard errors, inflating t-statistics and making variables appear significant when they are not. Heteroscedasticity makes confidence intervals too narrow or wide depending on the predictor region. Non-normal residuals affect p-value validity in small samples but less so in large samples (n > 100). Multicollinearity inflates coefficient variances, making individual coefficient estimates unstable and uninterpretable even though the overall prediction may still be good. + +**Q: Explain the difference between R² and Adjusted R². When would they diverge significantly?** +A: R² measures the proportion of variance in y explained by the model. It always increases (or stays the same) when you add more features, even if those features are pure noise. Adjusted R² applies a penalty based on the number of predictors and sample size: Adj_R² = 1 - (1 - R²)(n - 1)/(n - p - 1). When adding noise features, R² may tick up slightly while Adjusted R² decreases, signaling the feature is not useful. They diverge most when you have many irrelevant features relative to sample size. Rule of thumb: use Adjusted R² for model selection, R² for interpretation of explained variance. + +**Q: When would you choose Ridge over Lasso, and vice versa?** +A: Choose Ridge when you have many features that all likely contribute to the prediction — Ridge shrinks coefficients smoothly without eliminating them, preserving all features. Choose Lasso when you suspect only a subset of features are actually relevant — Lasso drives irrelevant coefficients to exactly zero, performing automatic feature selection. In practice, ElasticNet (combining both) often works best. For example, in a gene expression study with 10,000 genes but only 50 truly relevant, Lasso would zero out 9950 coefficients. In a marketing mix model where all channels contribute somewhat, Ridge keeps everything in. Use cross-validation to select α and the L1 ratio. + +### Related Concepts +Logistic Regression, Regularization, MSE + +--- + +## Logistic Regression +**Definition:** Binary classification via sigmoid function, minimizing log loss. Probabilistic output. + +### Theory & Explanation (3-5 paragraphs) + +Logistic regression models the probability that an instance belongs to the positive class using the sigmoid (logistic) function: P(y=1|X) = 1 / (1 + e^(-z)), where z = w₀ + w₁x₁ + ... + wₚxₚ. This squeezes the linear combination z from (-∞, +∞) into a probability between 0 and 1. The decision boundary occurs where P = 0.5, corresponding to z = 0. Unlike linear regression, logistic regression has no closed-form solution — coefficients are estimated by minimizing log loss (also called binary cross-entropy): L = -Σ[y_i·log(p_i) + (1-y_i)·log(1-p_i)], typically using gradient descent or Newton-Raphson optimization. + +The coefficients in logistic regression are interpreted in terms of log-odds. For a one-unit increase in xⱼ, the log-odds of the positive class change by wⱼ, holding other variables constant. Exponentiating gives the odds ratio: e^(wⱼ). For example, if wⱼ = 0.5, a one-unit increase multiplies the odds of the positive outcome by e^0.5 ≈ 1.65. This interpretability is why logistic regression remains immensely popular in consulting, banking, healthcare, and any domain where stakeholders need to understand "why" a prediction was made. + +Regularization is as important in logistic regression as in linear models. L2 regularization (ridge) is standard for handling separation or multicollinearity among predictors. L1 regularization (lasso) performs feature selection. In high-dimensional settings, regularized logistic regression (often with elastic net) is the go-to baseline before trying any black-box models. The regularization strength C in scikit-learn is the inverse of λ — smaller C means stronger regularization. + +For multi-class classification, logistic regression extends via two strategies: **One-vs-Rest (OvR)** trains K binary classifiers (each class vs all others) and picks the class with the highest confidence; **Softmax (Multinomial)** regression generalizes the sigmoid to K classes using the softmax function, producing a proper probability distribution across all classes. Softmax is preferred when classes are mutually exclusive; OvR works well even when they are not. + +### Example + +**Loan default prediction:** z = -2.5 + 0.03 × credit_score - 0.05 × DTI_ratio. For a borrower with credit score = 700 and DTI = 30: z = -2.5 + 0.03(700) - 0.05(30) = -2.5 + 21 - 1.5 = 17. P(default) = 1 / (1 + e^(-17)) ≈ 0.00000004 — essentially zero probability of default. The coefficient interpretation: each 1-point increase in credit score multiplies default odds by e^(0.03) = 1.03 (3% increase), while each 1-point increase in DTI multiplies default odds by e^(-0.05) = 0.95 (5% decrease). Model achieves AUC = 0.82 on test set with log loss = 0.31. + +### Interview Questions (2-3) + +**Q: Why is it called "regression" when it's used for classification?** +A: Historically, logistic regression grew out of linear regression — you're still regressing the log-odds of the outcome against the predictors. The classification step (applying a threshold to the probability) is secondary to the underlying regression on the log-odds scale. The name stuck. More practically, logistic regression estimates a continuous probability first; assigning a discrete class label is a post-processing decision based on a threshold. + +**Q: How do you determine the decision boundary and choose the optimal probability threshold?** +A: The default decision boundary is P = 0.5, but this is often suboptimal for imbalanced problems. The threshold should be chosen based on the business cost matrix. For fraud detection where false negatives cost $1000 and false positives cost $10, you'd lower the threshold to catch more fraud (trading precision for recall). The optimal threshold is found by computing expected cost across thresholds using the validation set, or by finding the point on the ROC curve closest to (0,1), or using the Youden index (maximizing sensitivity + specificity - 1). + +**Q: How do you handle multi-class classification with logistic regression?** +A: Two main approaches: One-vs-Rest (OvR) trains K separate binary classifiers, each distinguishing one class from all others. Prediction is the class whose classifier gives the highest confidence score. Softmax (multinomial) logistic regression generalizes the sigmoid to the softmax function, producing a proper probability distribution across all K classes. Softmax is preferred for mutually exclusive classes (e.g., digit 0-9) because the probabilities sum to 1. OvR works better when classes are not mutually exclusive (e.g., tagging an image with multiple labels) or when you need class-specific thresholding. + +### Related Concepts +Linear Regression, Sigmoid, Log Loss + +--- + +## Decision Trees +**Definition:** Recursive partitioning of feature space via if-then-else rules. Interpretable, no scaling needed. + +### Theory & Explanation (3-5 paragraphs) + +Decision trees split the data recursively based on feature values, creating a hierarchical structure of if-then-else rules. At each node, the algorithm searches over all features and all possible split points to find the split that most reduces impurity in the child nodes. For classification, impurity is measured by **Gini impurity** (G = 1 - Σp_k², where p_k is the proportion of class k in the node) or **entropy** (H = -Σp_k·log₂(p_k)). For regression, the criterion is MSE (minimizing within-node variance). The tree grows until a stopping criterion is met — typically a maximum depth or a minimum number of samples per leaf. + +This greedy recursive binary splitting produces highly interpretable models — you can literally print the tree and explain it to a non-technical stakeholder. Decision trees also require no feature scaling (they split on raw thresholds), handle both numerical and categorical features naturally, capture non-linear relationships automatically, and are robust to outliers. These properties make decision trees a favorite first try in consulting engagements where model explainability is paramount. + +The major drawback is high variance — decision trees are prone to overfitting. A small change in the training data can produce a completely different tree structure. This is because the greedy splitting algorithm makes locally optimal decisions that may not be globally optimal, and because trees can grow until every leaf contains a single instance. Two strategies combat this: **pre-pruning** stops tree growth early (max_depth, min_samples_leaf, min_samples_split, max_features), and **post-pruning** grows the tree fully then cuts back branches. Cost-complexity pruning (ccp_alpha) is a principled post-pruning method that penalizes tree size using a complexity parameter (α), finding the subtree that minimizes R(T) + α|T|. + +Feature importance from decision trees is a key output — computed as the total reduction in impurity (weighted by samples) contributed by each feature across all splits. This gives a ranked list of which features drive predictions, often surprising to domain experts and useful for feature engineering. For regression trees, the prediction is simply the mean target value in each leaf. Decision trees can also naturally handle missing data through surrogate splits, though this is less commonly used in practice compared to imputation. + +### Example + +**Loan approval tree:** Root: credit_score ≤ 650? If yes → DTI ≤ 40%? If yes → income ≥ $50K? If yes → approve (75% purity, 120 samples). If income < $50K → reject (80% purity, 40 samples). If DTI > 40% → reject outright (90% purity, 60 samples). If credit_score > 650 → approve with income check (85% purity, 200 samples). Max depth = 3, min_samples_leaf = 20. The tree achieves 83% accuracy on the test set. Top feature: credit_score (importance 0.52), followed by DTI (0.31), income (0.17). The tree reveals an explicit, auditable decision policy — valuable for regulatory compliance. + +### Interview Questions (2-3) + +**Q: What is the difference between Gini impurity and entropy? Which one should you use?** +A: Both measure node impurity for classification splits. Gini = 1 - Σp_k² ranges from 0 (pure) to 0.5 (balanced binary). Entropy = -Σp_k·log₂(p_k) ranges from 0 to 1. Entropy is slightly more computationally expensive (involves logarithms) and tends to produce slightly more balanced splits. In practice, the difference is negligible — both produce very similar trees. The scikit-learn default is Gini because it's faster to compute. A rule of thumb: use Gini for speed on large datasets, use entropy when you need slightly more balanced trees for interpretability (entropy penalizes impure nodes more heavily). + +**Q: Why do decision trees overfit and how do you control it?** +A: Decision trees overfit because they can grow until each leaf contains a single training instance, effectively memorizing the training data including its noise. The greedy recursive splitting mechanism makes locally optimal choices that compound errors deeper in the tree. Variance is high — different training samples produce wildly different tree structures. Control methods include: pre-pruning (max_depth=3-10, min_samples_leaf=20-50, min_samples_split=50-100), post-pruning with cost-complexity pruning (ccp_alpha tuned via cross-validation), limiting max_features at each split, and using ensemble methods (Random Forest, Gradient Boosting) as the primary defense. For tabular data, single trees are rarely used without ensembling. + +**Q: How do decision trees handle numerical vs categorical features differently?** +A: For numerical features, the algorithm sorts the values and evaluates splits at the midpoint between each consecutive pair of distinct values. For categorical features, the algorithm evaluates splits on subsets of categories — for a K-level categorical, there are 2^(K-1) - 1 possible binary partitions, so scikit-learn uses a greedy approximation. High-cardinality categoricals (e.g., zip code with 1000+ values) can cause overfitting, and trees may struggle to find the optimal split. For such features, target encoding followed by treating as numeric often works better. One-hot encoding creates many binary features that trees can split on individually. + +### Related Concepts +Random Forest, Gradient Boosting, Gini + +--- + +## Random Forest +**Definition:** Ensemble of decision trees trained on bootstrapped samples + random feature subsets. Reduces variance without increasing bias. + +### Theory & Explanation (3-5 paragraphs) + +Random Forest builds a large number of decision trees (typically 100-1000) and averages their predictions. Each tree is trained on a **bootstrap sample** of the training data — drawn with replacement, each containing about 63% of unique observations (the rest are out-of-bag, OOB). Additionally, at each split, only a random subset of features is considered (√p for classification, p/3 for regression). This dual randomness (row sampling + column sampling) decorrelates the trees, making the ensemble far more robust than any individual tree. + +The magic of Random Forest lies in how it reduces variance without increasing bias. A single decision tree has low bias but high variance — it overfits. Individual tree predictions are noisy and vary significantly with small data changes. But if you train many trees on different bootstrapped samples and average them, the variance of the average is σ²/ρ where σ² is individual tree variance and ρ is the correlation between trees. By reducing ρ (through feature subsampling and bootstrapping), the ensemble achieves dramatically lower variance with essentially the same bias. This is why Random Forest works so "out of the box" — it's remarkably robust to hyperparameter choices. + +One of Random Forest's best features is the built-in **out-of-bag (OOB) error**. Since each tree is trained on only ~63% of data, the remaining 37% can serve as a validation set for that tree. Aggregating OOB predictions across all trees gives an unbiased estimate of generalization error — essentially free cross-validation without needing a separate validation set. OOB error correlates well with test-set error and can be used for feature selection and hyperparameter tuning. + +Feature importance from Random Forest is more reliable than from a single tree because it's averaged across many trees, smoothing out the high variance of individual tree importance scores. Permutation importance (measuring the drop in score when shuffling a feature) is considered more robust than impurity-based importance, especially when features have different scales or cardinalities. Random Forest also naturally handles missing values, mixed data types, non-linear relationships, and interactions — it's often the first algorithm data scientists try on tabular data after establishing a linear baseline. + +### Example + +**Fraud detection:** 500 trees trained on 50,000 transactions with 30 features. Each tree uses √30 ≈ 5 features per split. OOB accuracy = 97.5%, OOB recall for fraud class = 82%. Top 3 features by permutation importance: transaction_amount (0.18), transaction_velocity (0.12), distance_from_home (0.09). The model achieves test-set AUC = 0.96 with minimal tuning (just set n_estimators=500, max_depth=None). Compared to a single decision tree (AUC = 0.88, test accuracy drops 8% from train), Random Forest generalizes far better. Feature subsampling (max_features='sqrt') proved critical — without it, tree correlation was too high and AUC dropped to 0.93. + +### Interview Questions (2-3) + +**Q: Why does Random Forest work so well without extensive hyperparameter tuning?** +A: Random Forest is remarkably robust because of the bias-variance decomposition. Individual trees have low bias but high variance. By averaging hundreds of decorrelated trees (achieved through bootstrap sampling and random feature subsets), the ensemble dramatically reduces variance while leaving bias unchanged. The hyperparameters (n_estimators, max_depth, etc.) have wide acceptable ranges — more trees always help (diminishing returns after ~200), and deeper trees are safe because the ensemble's averaging handles the overfitting. The key knob is max_features (√p default works well) and min_samples_leaf (1 default is usually fine). Unlike boosting, RF doesn't sequentially amplify errors, so early stopping isn't needed. This robustness makes RF the quintessential "try first" model. + +**Q: What is OOB error and how does it compare to cross-validation?** +A: OOB (out-of-bag) error is computed from the ~37% of training samples that each tree never sees during training. For each training sample, we collect predictions from all trees where that sample was OOB, aggregate them, and compute the error across all samples. This gives an unbiased estimate of generalization error without holding out a separate validation set or performing k-fold cross-validation. OOB error is essentially free — it comes from the bootstrap process itself. Studies show OOB error correlates very well with test-set error and is slightly more pessimistic than k-fold CV (since OOB samples are about 37% of data per tree vs 20% in 5-fold CV). OOB is computationally cheaper than CV but slightly less stable. + +**Q: When would you choose Random Forest over XGBoost?** +A: Choose Random Forest when: (1) your dataset is small to medium and you want results fast with minimal tuning — RF has far fewer hyperparameters; (2) you need robustness to outliers and noisy labels — boosting can overfit to outliers since it iteratively focuses on hard examples; (3) interpretability via feature importance is sufficient and you don't need the best possible accuracy; (4) your data has many irrelevant features — RF handles feature noise well; (5) training speed matters more than prediction speed — RF parallelizes trivially. Choose XGBoost when you need maximum accuracy, your data is large and clean, you have time for hyperparameter tuning, and you need to handle missing values natively or want the best competition-grade performance. + +### Related Concepts +Decision Trees, Bagging, Gradient Boosting + +--- + +## XGBoost +**Definition:** Optimized gradient boosting with L1/L2 regularization, sequential trees correcting errors. Dominates structured data. + +### Theory & Explanation (3-5 paragraphs) + +XGBoost (Extreme Gradient Boosting) builds an ensemble of decision trees **sequentially**, where each new tree tries to correct the errors made by all previous trees. While Random Forest trains trees independently in parallel, XGBoost trains them one at a time, each focusing on the residuals (or gradients) of the current ensemble. The algorithm uses gradient descent in function space — at each iteration, it fits a tree to the negative gradient of the loss function with respect to the current prediction. Conceptually: tree 1 models the target, tree 2 models the errors of tree 1, tree 3 models the errors of trees 1+2, and so on. The final prediction is the weighted sum of all trees. + +XGBoost introduced several key innovations that made it the dominant algorithm for structured/tabular data. First, it uses a **regularized objective**: L = Σl(y_i, ŷ_i) + ΣΩ(f_k), where Ω penalizes tree complexity via L2 on leaf scores and the number of leaves. This directly prevents overfitting, unlike earlier boosting implementations. Second, it uses **second-order (Newton-Raphson) optimization** — it uses both gradients and Hessians (second derivatives) to determine the optimal split and leaf values, converging faster than first-order methods. Third, the **approximate greedy split finding** algorithm bins continuous features into percentiles, then searches splits only at bin boundaries instead of every unique value, dramatically speeding up training. + +XGBoost handles missing values natively through **sparsity-aware learning** — during training, it learns a default direction for missing values (left or right child) for each split, treating missing as information rather than imputing. It also supports column subsampling (like RF), L1 and L2 regularization on weights, custom loss functions, and built-in cross-validation. The hyperparameter space is rich: n_estimators (number of trees), learning_rate (shrinkage, typically 0.01-0.3), max_depth (typically 3-10), subsample (row subsample, 0.5-1.0), colsample_bytree (feature subsample), gamma (minimum loss reduction for split), reg_alpha (L1), reg_lambda (L2), and min_child_weight (minimum sum of instance weight in child). + +However, XGBoost's power comes with a cost: it requires careful hyperparameter tuning. A default XGBoost with n_estimators=100 and max_depth=6 will overfit. Proper tuning involves first setting a reasonable learning rate and n_estimators (with early stopping), then tuning tree-specific parameters (max_depth, min_child_weight), then regularization (gamma, reg_alpha, reg_lambda), and finally reducing learning rate further. Early stopping on a validation set is essential — training stops when validation performance hasn't improved for N rounds, preventing overfitting and saving computation. + +### Example + +**House price prediction (Kaggle competition):** Dataset of 79 features, 1460 training samples. XGBoost: n_estimators=300, learning_rate=0.05, max_depth=6, subsample=0.8, colsample_bytree=0.8. Early stopping triggers at tree 250 (no improvement for 50 rounds). RMSE = $24,800 on test set vs Random Forest RMSE = $35,200 vs Linear Regression RMSE = $45,600. Feature importance shows top predictors: living area (0.21), overall quality (0.18), year built (0.09), total basement (0.07). Training time: 12 seconds for 250 trees (parallelized across CPU cores). RF training: 8 seconds for 500 trees. XGBoost predictions are 15% faster for inference due to fewer trees. + +### Interview Questions (2-3) + +**Q: Explain how gradient boosting works conceptually.** +A: Gradient boosting builds an ensemble sequentially. Start with a simple initial prediction (e.g., mean of target for regression). Calculate the residuals (actual - prediction) — these are proportional to the negative gradient of the loss function. Fit a shallow decision tree to predict these residuals. Add this tree's predictions (scaled by learning rate, typically 0.01-0.1) to the ensemble. Recalculate new residuals based on the updated ensemble. Fit the next tree to these new residuals. Repeat for M iterations. Each tree focuses on what the previous ensemble got wrong. The learning rate shrinks each tree's contribution — a lower rate requires more trees but generalizes better. Think of it as gradient descent in function space: each iteration takes a step in the direction that most reduces the loss. + +**Q: How does XGBoost handle missing values differently from other algorithms?** +A: XGBoost handles missing values natively through **sparsity-aware learning**. During training, for each split, XGBoost learns which direction (left or right child) missing values should go — it tries both directions and picks the one that gives the best loss reduction. This means missing values are treated as informative rather than something to impute. The default direction learned during training is used at prediction time. This is fundamentally different from most algorithms, which require imputation before training. XGBoost can handle data where 90%+ of values are missing (sparse matrices) natively. The sparsity-aware algorithm is also optimized for one-hot encoded features — it treats zeros as missing to skip unnecessary computations. + +**Q: What are the key differences between XGBoost and LightGBM?** +A: Four main differences: (1) **Tree growth strategy:** XGBoost grows trees level-wise (depth-wise), splitting all nodes at the current depth before moving deeper. LightGBM grows trees leaf-wise, splitting the leaf with the highest loss reduction first. Leaf-wise is faster and can find more complex patterns but is more prone to overfitting — LightGBM needs max_depth or num_leaves regularization. (2) **Split finding:** XGBoost uses pre-sorted or histogram-based algorithms. LightGBM uses Gradient-based One-Side Sampling (GOSS) — it keeps all high-gradient instances (hard to fit) and randomly samples low-gradient instances (already well-fit). This speeds up training significantly on large datasets. (3) **Categorical features:** LightGBM handles categorical features natively with optimal splitting, while XGBoost requires one-hot encoding. (4) **Performance:** LightGBM trains faster on very large datasets (100K+ rows) but can overfit on small data. XGBoost is more conservative and robust on small to medium datasets. In practice, LightGBM is growing in popularity but XGBoost remains the default choice for tabular competitions. + +### Related Concepts +Decision Trees, Random Forest, Gradient Boosting + +--- + +## K-Nearest Neighbors +**Definition:** Non-parametric lazy learner — votes of K nearest neighbors determine prediction. + +### Theory & Explanation (3-5 paragraphs) + +K-Nearest Neighbors (KNN) is one of the simplest machine learning algorithms — it memorizes the entire training dataset and makes predictions based on the similarity of new instances to stored examples. For classification, the predicted class is the majority vote among the K closest training points. For regression, the prediction is the mean (or median) target value of the K nearest neighbors. The algorithm requires no training phase (it's a "lazy learner"), which means training is instant — all computation happens at prediction time. + +The choice of K controls the bias-variance tradeoff. A small K (e.g., K=1) gives low bias but high variance — the prediction is highly sensitive to noise in the nearest single point. A large K (e.g., K=100) gives higher bias (predictions become smoother and lose local patterns) but lower variance. The optimal K is typically found through cross-validation, with odd values recommended for binary classification to avoid ties. Typical values range from 3 to 20, but optimal K depends on dataset size and noise level — larger datasets can support larger K values. + +Distance metric choice matters significantly. **Euclidean distance** (L2) is most common but assumes features are on the same scale. **Manhattan distance** (L1) is more robust to outliers. For high-dimensional data, distance metrics break down (the "curse of dimensionality"), and **cosine similarity** often works better for text or high-dim sparse data. Minkowski distance generalizes both L1 and L2 with a parameter p. Weighted KNN (where closer neighbors contribute more to the vote) is a common extension that improves performance by giving more influence to genuinely similar points. + +Feature scaling is absolutely critical for KNN. If one feature (e.g., income in dollars, range 0-1,000,000) has a much larger scale than another (e.g., age, range 0-100), the distance calculation will be dominated by the larger-scaled feature regardless of predictive importance. Standardization (z-score: subtract mean, divide by standard deviation) or normalization (min-max scaling to [0,1]) are essential preprocessing steps. Without scaling, KNN performs poorly on mixed-scale datasets. KD-trees and Ball trees are data structures that accelerate nearest neighbor search from O(n) to O(log n) in low dimensions, though they degrade in high dimensions where brute-force search may be optimal. + +### Example + +**Iris flower classification:** Dataset has 150 samples, 4 features (sepal length, sepal width, petal length, petal width), 3 species. With K=5 and Euclidean distance on standardized features, a new flower's 5 nearest neighbors are all setosa → predict setosa with probability 1.0. 5-fold cross-validation yields 97.3% accuracy. K=1 gives 98.7% train accuracy but 94% CV accuracy (overfitting). K=15 gives 96% CV accuracy (slightly high bias). Optimal K=5. Feature importance by dimension reduction: petal length and width dominate distance calculations — they separate species well even in 2D. + +### Interview Questions (2-3) + +**Q: Why is feature scaling critical for KNN?** +A: KNN relies entirely on distance calculations between points. If one feature (say, salary, range 0-200K) has a much larger numerical range than another (say, age, range 18-80), the distance will be dominated by salary regardless of whether age is more predictive. For example, two people with salaries 5K apart are considered very "distant" even if they have the same age. Standardization (z-score scaling) puts all features on equal footing — each contributes proportionally to the distance. Without scaling, KNN effectively ignores low-magnitude features. This is not optional for KNN — it's a hard requirement. The same applies to any distance-based algorithm: SVM with RBF kernel, K-Means, PCA. + +**Q: Explain the curse of dimensionality and its effect on KNN specifically.** +A: The curse of dimensionality refers to the phenomenon where distance metrics become meaningless in high-dimensional spaces. As dimensions increase, the volume of space grows exponentially, and points become nearly equidistant from each other — the ratio of nearest to farthest distances approaches 1. For KNN, this means all neighbors are approximately the same distance, making the K-nearest concept meaningless. Additionally, high-dimensional spaces are sparse — you need exponentially more training data to maintain the same density. For KNN, this means: (1) the number of training samples needed grows exponentially with dimensions (to have neighbors that are actually "close"), (2) distance metrics lose discriminative power, and (3) KD-trees and other acceleration structures perform worse than brute force. Mitigations include dimensionality reduction (PCA, feature selection), using L1 distance (which is more robust in high dimensions), or switching to similarity-based methods. + +**Q: Compare KNN's training vs prediction time complexity. Why does this matter in production?** +A: KNN has O(1) training time — it just stores the data (it's a lazy learner). However, prediction time is O(n×d) for brute force — it must compute distance between the new point and every training point for every prediction. With KD-trees, average prediction time can be O(log n) in low dimensions. This asymmetry matters enormously in production: KNN training is instant, but prediction latency grows linearly with dataset size. For a system with 10 million training samples and 1000 predictions/second, brute-force KNN would require 10 billion distance computations per second — infeasible. Mitigations include: using approximate nearest neighbor (ANN) algorithms (like Spotify's Annoy, Facebook's FAISS), limiting dataset size (sliding window of recent data), or dimensionality reduction to make KD-trees effective. In practice, KNN is rarely used in large-scale production systems without approximate neighbor search. + +### Related Concepts +Feature Scaling, Distance Metrics, Curse of Dimensionality + +--- + +## Support Vector Machine (SVM) +**Definition:** Finds max-margin hyperplane. Kernel trick enables non-linear boundaries. + +### Theory & Explanation (3-5 paragraphs) + +Support Vector Machine finds the hyperplane that maximizes the margin — the distance between the decision boundary and the nearest training points from each class. These nearest points are called **support vectors**, and they alone define the boundary. The maximum margin principle provides the strongest possible generalization guarantee: the margin width bounds the VC dimension of the classifier, making SVM theoretically well-founded even in high-dimensional spaces. The optimization problem is convex (no local minima), ensuring a unique global solution. + +In practice, data is rarely perfectly separable, so the **soft-margin formulation** introduces the C parameter, which controls the penalty for misclassifications. Large C: hard margin (tries to classify everything correctly, may overfit). Small C: soft margin (allows some misclassifications for a wider, more generalizable margin). C is the primary regularization parameter in SVM and is typically tuned via cross-validation on a logarithmic scale (e.g., 0.01, 0.1, 1, 10, 100). + +The **kernel trick** is SVM's superpower. Instead of explicitly computing features in a high-dimensional space (which could be computationally prohibitive), the kernel trick computes dot products in that space implicitly through a kernel function that operates on the original features. The **RBF (Radial Basis Function)** kernel, K(xᵢ, xⱼ) = exp(-γ||xᵢ - xⱼ||²), is the default choice. It maps data to an infinite-dimensional space and can approximate any non-linear boundary. The γ parameter controls the influence radius of each training point — small γ means broad influence (smooth boundary, high bias), large γ means each point influences only its immediate neighborhood (wavy boundary, high variance). + +SVM performs well on high-dimensional data (e.g., text classification with thousands of features) because the max-margin principle is less prone to overfitting in high dimensions than other algorithms. It's also memory-efficient — only the support vectors (often a small fraction of training data) need to be stored. However, SVM doesn't scale well to very large datasets (training complexity is O(n²) to O(n³) in the number of samples). It also doesn't provide probabilistic outputs natively — Platt scaling (fitting a logistic regression on SVM outputs) is needed for calibrated probabilities. SVM requires feature scaling (important for RBF kernel), and choosing the right kernel + hyperparameters (C, γ) requires careful grid search or Bayesian optimization. + +### Example + +**Handwritten digit classification (8×8 pixel images = 64 features):** RBF kernel, γ = 0.001, C = 10. Training on 1347 samples → 412 are support vectors (31%). Test accuracy = 98.5%. Confusion matrix shows 2→3 and 4→9 are the most common misclassifications. Grid search shows C=10, γ=0.001 is near-optimal. A linear SVM achieves 96.2% (showing the non-linear boundary helps), and Random Forest achieves 97.8%. SVM's memory footprint is small — only 412 support vectors × 64 dimensions need storing, vs RF which stores 500 trees. Training time: 3.2 seconds vs RF's 1.5 seconds. + +### Interview Questions (2-3) + +**Q: What is the role of the C parameter in SVM?** +A: C controls the trade-off between achieving a wide margin and minimizing training errors. High C (hard margin) penalizes misclassifications heavily — the algorithm will sacrifice margin width to classify every training point correctly. This can lead to overfitting if there are outliers. Low C (soft margin) allows some training points to be misclassified or fall within the margin in exchange for a wider margin. The wider margin leads to better generalization on test data but risks underfitting if C is too low. C is analogous to 1/λ in regularized regression. In practice, C is tuned via cross-validation, typically on a logarithmic scale (10^k where k ranges from -3 to 3). + +**Q: Explain the kernel trick in simple terms.** +A: The kernel trick allows SVM to find non-linear decision boundaries without explicitly computing features in a high-dimensional space. Intuitively, imagine data points arranged in a circle (one class inside, another outside) in 2D — no linear classifier can separate them. But if you map each point to 3D using (x, y) → (x, y, x²+y²), the points become linearly separable in 3D by a plane cutting at the right height. The kernel trick computes the dot product in that 3D space without ever doing the mapping explicitly — the RBF kernel K(x,y) = exp(-γ||x-y||²) implicitly computes dot products in an infinite-dimensional space using only the original 2D coordinates. This means SVM can learn complex boundaries at the computational cost of the kernel function rather than the cost of the high-dimensional feature space. + +**Q: When would you choose SVM over Random Forest?** +A: Choose SVM when: (1) you have high-dimensional data with relatively few samples (e.g., text classification with 50K features and 1K documents) — SVM's max-margin principle generalizes well in high dimensions; (2) interpretability via support vectors is valuable; (3) you need a convex, unique solution that doesn't depend on random initialization; (4) your data has clear class separation and a medium-size dataset (up to ~50K samples) — SVM is computationally expensive for large datasets. Choose Random Forest when: (1) your dataset is large (100K+ samples); (2) you have mixed data types (numeric + categorical) with complex interactions; (3) you need feature importance scores; (4) you want something that works well out of the box with minimal tuning. In practice, on tabular data, RF and XGBoost have largely replaced SVM for most problems, but SVM remains excellent for text, images (pre-deep-learning), and small-to-medium high-dim datasets. + +### Related Concepts +Kernel Trick, Support Vectors, Regularization + +--- + +## K-Means Clustering +**Definition:** Partitions data into k clusters minimizing within-cluster variance through iterative centroid refinement. + +### Theory & Explanation (3-5 paragraphs) + +K-Means is the most widely used clustering algorithm due to its simplicity and speed. It partitions n observations into k clusters, each represented by its centroid (the mean of points in the cluster). The algorithm alternates between two steps: (1) **assignment** — assign each point to the nearest centroid, and (2) **update** — recompute centroids as the mean of all points assigned to each cluster. This continues until centroids stabilize (or a maximum number of iterations is reached). The objective (inertia) is to minimize the within-cluster sum of squares: Σᵢ min_c ||xᵢ - μ_c||². + +The algorithm is sensitive to initialization — poor initial centroids can lead to suboptimal clusters or empty clusters. **K-Means++** initialization selects initial centroids probabilistically: the first centroid is randomly chosen, and subsequent centroids are selected with probability proportional to the squared distance from the nearest existing centroid. This spreads initial centroids across the data and significantly improves both convergence speed and solution quality. Even with K-Means++, multiple random restarts (n_init=10-25) are standard practice. + +Choosing the number of clusters K is the fundamental challenge. The **elbow method** plots inertia vs K and looks for the "elbow" where adding more clusters yields diminishing returns. The **silhouette score** (ranging from -1 to 1) measures how similar a point is to its own cluster vs neighboring clusters — the optimal K typically maximizes the average silhouette score. Domain knowledge is often the most important guide. Gap statistics and the Davies-Bouldin index are additional criteria. There is rarely a single "correct" K — multiple K values should be evaluated for business interpretability. + +K-Means has several well-known limitations. It assumes clusters are **spherical** (isotropic) and **roughly equal in size** — it fails on elongated, nested, or irregularly shaped clusters (DBSCAN handles these better). It's sensitive to **feature scaling** — without standardization, high-magnitude features dominate the distance calculation. It's sensitive to **outliers** — outliers pull centroids toward them. It only finds **convex clusters** and cannot separate non-convex clusters (like concentric circles). It's also **non-deterministic** — different random starts can produce different results. Despite these limitations, K-Means remains the first clustering algorithm to try because it's fast (O(n × k × d × I)), interpretable, and works well when assumptions are reasonably met. + +### Example + +**RFM customer segmentation for e-commerce:** 10,000 customers characterized by Recency (days since last purchase), Frequency (number of purchases), Monetary value (total spend). After standardization, K=4 clusters identified: **VIP** (15%): high R, F, M — highest value, most recent. **Bargain hunters** (35%): moderate F, low M — buy often but spend little. **Big spenders** (10%): low F, very high M — rare but expensive purchases. **At risk** (40%): high R, low F, low M — likely churning. Silhouette score = 0.58 (reasonable structure). Elbow plot shows K=4 as knee point. Business action: VIP retention program, re-engagement campaign for At risk, upsell to Bargain hunters. + +### Interview Questions (2-3) + +**Q: How do you choose the optimal number of clusters K in K-Means?** +A: Multiple complementary approaches: (1) **Elbow method** — plot inertia (within-cluster SSE) vs K. Choose K where the rate of decrease sharply changes (the "elbow"). This is subjective but widely used. (2) **Silhouette score** — compute average silhouette width across K values; choose K that maximizes it. Silhouette ranges from -1 to 1 and measures both cohesion and separation. (3) **Gap statistic** — compares within-cluster dispersion to that expected under a null reference distribution; choose K where the gap is largest. (4) **Domain knowledge** — the business context may dictate K (e.g., budget for 3 customer segments). (5) **Stability analysis** — run K-Means with different random seeds and check if cluster assignments are consistent. In practice, compute 3-4 criteria and look for consensus. For customer segmentation, K=4 or K=5 is a common starting point. + +**Q: What are the main limitations of K-Means clustering?** +A: Six key limitations: (1) **Assumes spherical clusters** — K-Means minimizes Euclidean distance, which implicitly assumes clusters are spherical and isotropic. It fails on elongated, crescent-shaped, or nested clusters. (2) **Sensitive to scale** — features must be standardized or the largest-scale feature dominates. (3) **Sensitive to outliers** — outliers pull centroids toward them, distorting all clusters. Pre-filter outliers or use K-Medoids. (4) **Requires K to be specified** — there's no built-in way to determine K automatically. (5) **Non-deterministic** — different initializations give different results. Use K-Means++ and multiple restarts. (6) **Assumes all clusters are equally important** — it tends to produce clusters of similar size (due to the equal weighting of points). If your data has natural clusters of very different sizes or densities, K-Means will split large clusters and merge small ones. For such data, consider Gaussian Mixture Models (which handle varying cluster sizes and shapes via covariance structure) or DBSCAN. + +**Q: Compare K-Means and Gaussian Mixture Models (GMM).** +A: K-Means is a special case of GMM with hard assignments and spherical clusters. GMM is a probabilistic model where each cluster is a multivariate Gaussian distribution with its own mean and covariance matrix. Key differences: (1) **Assignments** — K-Means gives hard assignments (each point belongs entirely to one cluster); GMM gives soft assignments (each point has a probability of belonging to each cluster). (2) **Cluster shape** — K-Means assumes spherical clusters; GMM can model ellipsoidal clusters of different sizes and orientations through the covariance type (full, tied, diag, spherical). (3) **Robustness** — GMM is more robust to noise and varying cluster sizes because it models density rather than distance. (4) **Computational cost** — K-Means is O(n×k×d), GMM is more expensive (requires E-step and M-step with covariance matrix inversion). (5) **Convergence** — K-Means may converge to local optima; GMM has similar issues. Use K-Means for initial exploration and when clusters are well-separated and spherical. Use GMM when clusters have different shapes, sizes, or orientations. + +### Related Concepts +Unsupervised Learning, DBSCAN, Hierarchical Clustering + +--- + +## PCA (Principal Component Analysis) +**Definition:** Unsupervised dimensionality reduction transforming features to orthogonal components capturing maximum variance. + +### Theory & Explanation (3-5 paragraphs) + +Principal Component Analysis (PCA) finds a set of orthogonal axes (principal components) that capture the maximum variance in the data. The first principal component is the direction of greatest variance. The second is orthogonal to the first and captures the next greatest variance, and so on. Mathematically, PCA finds the eigenvectors of the covariance matrix Σ = (1/n)XᵀX (for centered data). The eigenvalues represent the variance explained by each component. The principal components are the eigenvectors, sorted by descending eigenvalue. Dimensionality reduction is achieved by projecting the data onto the top k components, discarding the rest. + +PCA serves multiple purposes. **Visualization:** projecting high-dimensional data to 2D or 3D for exploratory analysis. **Noise reduction:** the last components (smallest eigenvalues) often capture noise; dropping them improves downstream model performance. **Multicollinearity handling:** correlated features collapse into a few PCs, making regression more stable. **Computational efficiency:** fewer features mean faster training. **Compression:** storing data in PCA space requires fewer numbers. The price is interpretability — each PC is a linear combination of all original features, making it hard to say what a PC "means" in domain terms. + +Feature scaling is absolutely essential before PCA. If variables are on different scales (e.g., income in dollars vs age in years), the variance will be dominated by the large-scale variable, and PCA will effectively ignore everything else. Standardization (z-score) is recommended unless all features are on the same scale. Some practitioners prefer using the correlation matrix (which is the covariance matrix of standardized data) vs the covariance matrix to automatically account for scale differences. + +The explained variance ratio (λᵢ / Σλⱼ) tells us the proportion of total variance captured by each component. A common rule of thumb is to retain enough components to explain 90-95% of variance. The **scree plot** (eigenvalues vs component index) shows the "elbow" where additional components contribute little. However, for downstream ML tasks, sometimes fewer components (capturing less variance) work better — the "signal" may be concentrated in the first few components while noise fills the rest. PCA is a linear method — it cannot capture non-linear structures (use t-SNE, UMAP, or autoencoders for that). It also makes the assumption that high variance = important signal, which is not always true (a feature with low variance could be the most predictive one). + +### Example + +**Survey data reduction:** Customer satisfaction survey with 50 highly correlated features. PCA applied on standardized features. Results: PC1 (28% variance) — overall satisfaction (all features load positively). PC2 (18%) — service speed vs thoroughness (positive loadings on speed items, negative on detail items). PC3 (12%) — digital vs in-person satisfaction. Top 15 PCs capture 93% of variance. Logistic regression on original 50 features: AUC = 0.72 (overfits, with 50 features for 500 samples). On 15 PCs: AUC = 0.81 (better generalization, stable coefficients). Component loadings reveal that PC1 represents the "halo effect" — customers who rate one thing highly tend to rate everything highly. + +### Interview Questions (2-3) + +**Q: Why must features be scaled before applying PCA?** +A: PCA finds directions of maximum variance. If features are on different scales (e.g., salary spanning 20K-200K vs age spanning 20-80), the variance will be dominated by the large-scale feature(s). A direction that captures salary variance will have a much higher eigenvalue than one capturing age variance — simply because of the unit difference, not because salary is more important. Standardizing (z-score) puts all features on equal footing so variance reflects informativeness rather than scale. Without scaling, PCA on mixed-scale data is effectively just working with the two or three largest-scale features. The only exception is when all features are on the same scale and unit (e.g., pixel intensities 0-255 in image data). + +**Q: How do you interpret principal components in business terms?** +A: Principal components are linear combinations of all original features, making direct interpretation challenging. The interpretation is based on examining **loadings** (the coefficients of each original feature on each PC). A PC with large positive loadings on several correlated features and near-zero on others represents a "composite" of those features. For example, in survey data, PC1 might have high positive loadings on all satisfaction questions → it's an "overall satisfaction" index. PC2 might have positive loadings on "speed" questions and negative on "thoroughness" → it represents the speed-vs-quality tradeoff. Techniques to aid interpretation: use **varimax rotation** (simplifies loadings), examine the top features contributing to each PC, or use sparse PCA (which drives small loadings to zero). For reporting, you can name each PC based on its dominant theme (e.g., "Service Quality Factor", "Price Sensitivity Factor"). + +**Q: What are the limitations of PCA?** +A: Four key limitations: (1) **Linearity** — PCA only finds linear combinations. For data with non-linear structure (e.g., a curved manifold), PCA will mix independent patterns across multiple components. Use kernel PCA, t-SNE, or UMAP for non-linear reduction. (2) **Variance ≠ importance** — PCA assumes maximum variance corresponds to maximum information. This can fail: a feature with low variance may be the most predictive one for your task (picture a categorical feature where one rare class is the key predictor). PCA may discard it entirely. (3) **Interpretability loss** — each PC is a weighted sum of all original features, making it hard to explain what each component represents to stakeholders. This matters in regulated industries. (4) **Outlier sensitivity** — a few extreme points can dramatically alter component directions. Robust PCA variants address this. PCA should never be applied blindly — always examine the loadings, explained variance, and impact on downstream task performance. + +### Related Concepts +Dimensionality Reduction, Feature Extraction, t-SNE + +--- + +## Gradient Boosting (General) +**Definition:** Sequential ensemble where each model corrects previous errors. Reduces bias. + +### Theory & Explanation (3-5 paragraphs) + +Gradient Boosting is a general framework for building predictive models by sequentially adding weak learners (usually shallow decision trees) that correct the errors of the existing ensemble. Unlike Random Forest (which builds trees independently and averages), gradient boosting builds trees **sequentially** — each new tree is trained to predict the **residuals** (or, more precisely, the negative gradient of the loss function) of the current ensemble. The first model (e.g., the mean of the target) makes a crude prediction. Tree 1 learns to predict the error of this first model. The ensemble becomes: F₁(x) = initial_prediction + lr × tree₁(x). Tree 2 learns to predict the error of F₁(x), and so on. + +The **learning rate** (also called shrinkage, typically 0.01-0.3) scales the contribution of each tree. A low learning rate means each tree contributes less, requiring more trees but generally producing a better-generalizing model. This creates a tradeoff: more trees + lower learning rate = better fit but longer training time and more risk of overfitting if not stopped early. The standard approach is to set a low learning rate (0.01-0.1) and use **early stopping** — monitor validation set performance and stop adding trees when it stops improving. + +Gradient boosting reduces **bias** primarily (unlike Random Forest which reduces variance). The first tree captures the broad patterns in the data; each subsequent tree focuses on harder-to-predict cases that earlier trees got wrong. This sequential error correction allows gradient boosting to achieve very high accuracy — often the best-performing algorithm on structured/tabular data. However, this power comes with overfitting risk — if you add too many trees or use deep trees, the model starts memorizing noise in the training data. + +Key hyperparameters control the bias-variance tradeoff: **n_estimators** (number of trees — increase until validation error stops dropping), **max_depth** (typically 3-8 — shallow trees are weak learners that generalize better), **learning_rate** (shrinkage — lower values need more trees), **subsample** (row sampling, 0.5-1.0 — stochastic gradient boosting introduces randomness to prevent overfitting), and **min_samples_leaf** (prevents leaves with too few samples). The interaction between these parameters is complex — for example, reducing the learning rate usually requires increasing n_estimators and can allow deeper trees before overfitting begins. + +**Bagging vs Boosting** is a fundamental distinction in ensemble learning. Bagging (Bootstrap Aggregating, used in Random Forest) trains models independently in parallel on bootstrapped data subsets and averages their predictions. It primarily reduces variance. Boosting trains models sequentially, each focusing on the errors of the previous ones. It primarily reduces bias. Bagging is simpler, more robust, and needs less tuning. Boosting is more complex, needs more tuning, but can achieve higher accuracy. In practice, try Random Forest first (quick, robust), then try gradient boosting to squeeze out extra performance. + +### Example + +**Customer churn prediction:** Dataset of 50,000 telecom customers with 20 features. Gradient boosting with 200 trees, max_depth=4, learning_rate=0.05, subsample=0.8. Test AUC = 0.87. Comparison: Logistic Regression AUC = 0.78 (linear boundary inadequate), Random Forest AUC = 0.83 (good but boosting does better), XGBoost AUC = 0.88 (slightly better due to regularization). Early stopping triggers at tree 180 (no improvement for 20 rounds). Feature importance: contract_type (0.22), tenure (0.18), monthly_charges (0.14), tech_support (0.09). Partial dependence plots show churn risk drops sharply in first 12 months of tenure. Training time: 45 seconds for 200 trees (sequential, hard to parallelize). + +### Interview Questions (2-3) + +**Q: Explain the difference between bagging and boosting.** +A: Bagging (Bootstrap Aggregating, e.g., Random Forest) trains base models independently in parallel on random bootstrap samples of the data. Predictions are averaged (regression) or voted (classification). Bagging primarily reduces **variance** — each model overfits differently, and averaging cancels out individual overfitting. Boosting trains base models **sequentially**, where each new model focuses on the mistakes made by the previous ensemble. Boosting primarily reduces **bias** — the ensemble starts simple and progressively learns harder patterns. Analogy: bagging is a committee of independent experts voting; if one expert is wrong, others compensate. Boosting is an iterative improvement process where each stage addresses weaknesses of the prior stages. Bagging is parallelizable, robust, and hard to overfit. Boosting is sequential, more prone to overfitting, but can achieve higher accuracy. + +**Q: How do you tune the learning rate in gradient boosting?** +A: The learning rate shrinks each tree's contribution. Lower learning rates (0.01-0.1) require more trees but generalize better. The tuning strategy: (1) Set a moderately low learning rate (0.1) and find the optimal number of trees via early stopping on a validation set. (2) Reduce the learning rate (e.g., 0.05 or 0.01) and increase n_estimators proportionally (e.g., if you halve the learning rate, double the number of trees). (3) Continue until further reductions don't improve validation performance. (4) Grid search or Bayesian optimization can find the optimal combination. Rule of thumb: learning_rate × n_estimators ≈ constant for the same validation fit (roughly). A common production recipe: learning_rate=0.01, n_estimators=1000-3000, early_stopping_rounds=50. Very low learning rates (0.001) with many trees (10,000+) can produce the best results but require substantial computation. + +**Q: How do you prevent overfitting in gradient boosting?** +A: Multiple complementary strategies: (1) **Early stopping** — monitor validation set performance and stop adding trees when it stops improving (typically after N rounds of no improvement). This is the single most important technique. (2) **Limit tree complexity** — keep max_depth shallow (3-6), enforce min_samples_leaf (20-50), limit min_child_weight. Shallow trees are "weak learners" that generalize better. (3) **Learning rate** — lower learning rates (0.01-0.05) require more trees but reduce the contribution of any single tree, limiting overfitting. (4) **Subsampling** — use stochastic gradient boosting (subsample=0.5-0.8) to add randomness, reducing the correlation between trees. (5) **Regularization** — L1 (reg_alpha) and L2 (reg_lambda) penalties on leaf scores directly penalize extreme predictions. (6) **Feature subsampling** (colsample_bytree) — similar to Random Forest, this decorrelates trees. (7) **Post-pruning** — apply cost-complexity pruning to individual trees. In practice, early stopping + shallow trees + learning rate tuning handles 90% of overfitting cases. + +### Related Concepts +Random Forest, XGBoost, Ensemble Methods + +--- + +## Evaluation Metrics (Classification & Regression) +**Definition:** Quantitative measures to assess model performance. Choice depends on business context and class balance. + +### Theory & Explanation (3-5 paragraphs) + +Classification metrics begin with the **confusion matrix** — the 2×2 table of True Positives, True Negatives, False Positives (Type I error), and False Negatives (Type II error). From this, we derive: **Accuracy** = (TP + TN) / (TP + TN + FP + FN) — the fraction of correct predictions. **Precision** = TP / (TP + FP) — of those predicted positive, how many are truly positive? **Recall (Sensitivity)** = TP / (TP + FN) — of those truly positive, how many did we catch? **Specificity** = TN / (TN + FP) — of those truly negative, how many did we correctly reject? **F1-score** = 2 × (Precision × Recall) / (Precision + Recall) — the harmonic mean of precision and recall, balancing both. + +For imbalanced classification (e.g., 1% fraud, 99% legitimate), accuracy is misleading — a model that always predicts "legitimate" achieves 99% accuracy but is useless. **Precision-Recall curves** show the tradeoff between precision and recall across thresholds. **AUC-ROC** (Area Under the Receiver Operating Characteristic curve) plots the true positive rate against the false positive rate across thresholds. AUC-ROC of 0.5 = random, 1.0 = perfect. However, for highly imbalanced data (especially when the minority class is of primary interest), **AUC-PR** (Area Under the Precision-Recall curve) is preferred — it focuses on the positive class and changes more dramatically when model performance on the rare class improves. + +Regression metrics evaluate how close predicted values are to actual values. **MSE** (Mean Squared Error) = (1/n)Σ(yᵢ - ŷᵢ)² — penalizes large errors more heavily, sensitive to outliers. **RMSE** = √MSE — in the same units as the target, more interpretable. **MAE** (Mean Absolute Error) = (1/n)Σ|yᵢ - ŷᵢ| — robust to outliers, linear penalty. **R²** = 1 - (SS_res / SS_tot) — proportion of variance explained; can be negative for terrible models. **Adjusted R²** penalizes feature count. **MAPE** (Mean Absolute Percentage Error) = (100%/n)Σ|(yᵢ - ŷᵢ) / yᵢ| — unitless, interpretable as "average % error", but undefined when yᵢ = 0 and asymmetric (penalizes overestimates more than underestimates). + +The choice of evaluation metric must align with the **business cost structure**. In fraud detection, a false negative costs $500 (lost fraud detection) while a false positive costs $20 (manual review time). The optimal threshold minimizes expected cost: Cost = FN × cost_FN + FP × cost_FP. This often means trading precision for recall or vice versa. **Cost-sensitive learning** incorporates these costs directly into model training (e.g., class weights in logistic regression, scale_pos_weight in XGBoost). The **lift curve** and **gains chart** are business-facing metrics that show how many positive cases the model captures at various deciles of predicted probability, directly translating to ROI calculations. + +### Example + +**Fraud detection (1% fraud rate, 100K transactions):** Confusion matrix at default 0.5 threshold: TN=98,700, FP=300, FN=200, TP=800. Accuracy = (98,700+800)/100,000 = 99.5%. Precision = 800/(800+300) = 72.7%. Recall = 800/(800+200) = 80%. F1 = 2×(0.727×0.80)/(0.727+0.80) = 0.76. AUC-ROC = 0.95 (excellent rank-ordering). AUC-PR = 0.42 (moderate — reflects the difficulty of the imbalanced task). Cost analysis: FN costs $500, FP costs $20. At default threshold: Cost = 200×$500 + 300×$20 = $106,000. Optimized threshold (P=0.15): TN=96,500, FP=2,500, FN=60, TP=940. Cost = 60×$500 + 2,500×$20 = $80,000 — saving $26,000 (25% reduction). The optimal threshold is far from 0.5 because FP cost is much lower than FN cost. + +### Interview Questions (2-3) + +**Q: Why is accuracy a poor metric for imbalanced classification?** +A: Accuracy treats all correct predictions equally, regardless of class. In a 99:1 imbalanced dataset, a trivial model that always predicts the majority class achieves 99% accuracy while completely failing at the business task (e.g., catching 0% of fraud, 0% of rare diseases). Accuracy does not distinguish between false positives and false negatives — which usually have vastly different costs. Better alternatives: precision-recall tradeoff (if the minority class matters), AUC-ROC (for rank-ordering ability), or cost-based metrics that incorporate the real business costs of different error types. For severely imbalanced data (99.9:0.1), even AUC-ROC can be misleadingly optimistic, and AUC-PR is preferred. + +**Q: Explain the precision-recall tradeoff. Can both be high simultaneously?** +A: Precision and recall have an inverse relationship controlled by the decision threshold. At a low threshold (say P=0.3), you classify more cases as positive — you catch more true positives (high recall) but also get more false positives (low precision). At a high threshold (P=0.9), you only predict positive when very confident — you have few false positives (high precision) but miss many true positives (low recall). Both can be high simultaneously only when the model perfectly separates the classes — the positive and negative distributions don't overlap. In real-world noisy data, there's always overlap, creating the tradeoff. The F1-score (harmonic mean) balances both, but the optimal operating point depends on business cost: high precision for spam detection (don't accidentally delete important email), high recall for cancer screening (don't miss a case). + +**Q: When should you use AUC-ROC vs AUC-PR?** +A: Use AUC-ROC when classes are reasonably balanced and false positive rate matters. AUC-ROC is the standard metric for general-purpose classification evaluation. It's insensitive to class imbalance in the sense that it stays the same regardless of the proportion of positives (if the model's rank-ordering ability doesn't change). Use AUC-PR when: (1) classes are highly imbalanced (positive class < 5%) — AUC-PR focuses on the positive class and changes more dramatically with performance changes on rare events; (2) the positive class is the primary focus (fraud, disease detection, churn); (3) you want to compare models on the practical region of interest (high-precision area). Brad Davis's rule of thumb: if the positive class constitutes < 20% of data and you care about positive class prediction, use PR curves. When you care about both classes equally and they're balanced, use ROC. In production, always report both plus the cost-based metric. + +### Related Concepts +Confusion Matrix, Imbalanced Data, F1-Score + +--- + +## LightGBM +**Definition:** Gradient boosting framework with leaf-wise tree growth, Gradient-based One-Side Sampling (GOSS), and Exclusive Feature Bundling (EFB). Faster training and lower memory than XGBoost with comparable accuracy. + +### Theory & Explanation (2-4 paragraphs) + +LightGBM differs from XGBoost primarily in tree growth strategy. While XGBoost grows trees **level-wise** (depth-first, balancing tree structure), LightGBM uses **leaf-wise** growth — at each step, it splits the leaf with the highest loss reduction, regardless of depth. This produces asymmetric trees that converge faster and can reach lower loss with fewer splits. However, leaf-wise growth is more prone to overfitting on small datasets, which is why LightGBM exposes the `num_leaves` parameter (typically 31 vs 2^max_depth in XGBoost) and `min_data_in_leaf` to control leaf-level regularization. The leaf-wise strategy enables LightGBM to achieve XGBoost-level accuracy with 10-20× faster training. + +Two algorithmic innovations drive LightGBM's speed. **GOSS (Gradient-based One-Side Sampling)** — instead of using all data points to estimate information gain, GOSS retains all instances with large gradients (under-fit points) and randomly subsamples instances with small gradients (well-fit points), with an importance weight to correct the sampling bias. This preserves accuracy while substantially reducing the number of data points evaluated at each split. **EFB (Exclusive Feature Bundling)** — high-dimensional data often has many mutually exclusive features (e.g., one-hot encoded categories never co-occur). EFB bundles such features together into a single feature, reducing the effective dimensionality and the number of split candidates without losing information. For a one-hot encoded dataset with 1000 categories, EFB can bundle them into a handful of dense features. + +LightGBM natively handles categorical features by grouping categories according to the target statistic (ordering categories by mean target value and finding optimal splits). This avoids one-hot encoding entirely, which XGBoost requires or handles less efficiently. LightGBM also supports GPU training with excellent out-of-the-box performance, histogram-based split finding (discretizing continuous features into bins for fast split search), and built handling of missing values. The API design prioritizes efficiency — the native `Dataset` format with pre-computed histograms means the first training pass is slower (building histograms) but subsequent training with different hyperparameters is fast. + +### Example + +**Click-through rate prediction (10M rows, 500 features, 1000 categoricals):** LightGBM trains in 3.5 minutes vs XGBoost's 28 minutes on the same hardware. Parameters: `objective=binary`, `num_leaves=63`, `learning_rate=0.05`, `min_data_in_leaf=50`, `feature_fraction=0.8`, `bagging_fraction=0.8`, `bagging_freq=5`. AUC on validation: 0.783 (LightGBM) vs 0.781 (XGBoost) — essentially identical accuracy at 8× training speed. With GOSS enabled (`top_rate=0.2, other_rate=0.1`), training drops to 2.1 minutes with AUC = 0.780. The `feature_fraction` (column subsampling) prevents overfitting on the high-dimensional feature set. Running on GPU (`device=gpu`) further reduces training to 45 seconds. + +### Interview Questions (2-3) + +**Q: When would you choose LightGBM over XGBoost?** +A: Choose LightGBM when: (1) training speed matters — LightGBM is typically 5-10× faster, especially on large datasets (100K+ rows). (2) You have high-cardinality categorical features — LightGBM handles them natively without one-hot encoding. (3) Memory is constrained — EFB reduces effective dimensionality. (4) You're running on GPU — LightGBM's GPU implementation is more mature and faster. Choose XGBoost when: (1) Dataset is small (< 10K rows) — LightGBM's leaf-wise growth overfits easily; XGBoost's level-wise growth is more robust. (2) You need the depth-first tree structure for interpretability. (3) You need the `monotone_constraints` feature (though LightGBM also supports this now). (4) Deep tree exploration matters — XGBoost's max_depth parameter is more intuitive than num_leaves. + +**Q: Explain the leaf-wise vs level-wise tree growth tradeoff.** +A: Level-wise growth (XGBoost) expands all leaves at the same depth before proceeding deeper, producing balanced trees. Leaf-wise growth (LightGBM) splits the leaf with the maximum loss reduction each time, producing asymmetric trees. Leaf-wise converges to lower loss with fewer nodes because it focuses computational effort on the most impactful splits. However, leaf-wise trees can grow very deep on one branch, capturing noise patterns specific to the training set, increasing overfitting risk. The `num_leaves` parameter directly controls the maximum number of leaves — setting it to 2^max_depth approximates a complete tree. A common recipe: `num_leaves=31` (equivalent to a depth-5 complete tree) with `min_data_in_leaf=20` for robustness against overfitting on small datasets. + +**Q: What are GOSS and EFB and why do they matter?** +A: GOSS (Gradient-based One-Side Sampling) is a data sampling technique that keeps all data points with large gradients (poorly predicted instances that contribute most to the loss) and randomly subsamples those with small gradients (well-predicted instances). This focuses training on the "hard" examples while using fewer total instances per iteration. GOSS has a theoretical guarantee that the estimated information gain differs from the true gain by at most a bounded error. EFB (Exclusive Feature Bundling) exploits the fact that high-dimensional sparse features (especially one-hot encoded categoricals) are often mutually exclusive — they rarely co-occur as non-zero. By bundling exclusive features into a single composite feature, EFB reduces the effective feature count from 1000+ to perhaps 10-20, dramatically accelerating histogram building and split search. EFB is lossless if features are truly mutually exclusive, and near-lossless otherwise. + +--- + +## Naive Bayes +**Definition:** Probabilistic classifier applying Bayes theorem with the strong (naive) assumption of conditional independence between features given the class. Fast, simple, effective for text classification. + +### Theory & Explanation (2-4 paragraphs) + +Naive Bayes applies Bayes theorem to compute the posterior probability of each class given the features: P(C_k | x) = P(C_k) × Π P(x_i | C_k) / P(x), where P(C_k) is the class prior, and the product Π P(x_i | C_k) is the likelihood of each feature given the class. The denominator P(x) is constant across classes and can be ignored for classification. The "naive" assumption is that features are conditionally independent given the class — P(x_i, x_j | C_k) = P(x_i | C_k) × P(x_j | C_k). This assumption is almost always violated in real data (words in a document are clearly not independent), yet Naive Bayes often performs surprisingly well, especially when the feature-class relationships are stronger than the feature-feature correlations. + +Three main variants handle different feature distributions. **Gaussian Naive Bayes** assumes features follow a normal distribution within each class — P(x_i | C_k) = N(x_i | μ_ik, σ²_ik). Used for continuous features like height, temperature, or sensor readings. **Multinomial Naive Bayes** models features as counts or frequencies, with P(x_i | C_k) proportional to the frequency of feature i in class C_k, smoothed by Laplace (α=1) or Lidstone (α<1) smoothing to handle zero counts. This is the standard for text classification with bag-of-words or TF-IDF features. **Bernoulli Naive Bayes** assumes binary features (word present/absent), with P(x_i | C_k) = P(x_i=1 | C_k) for the feature's presence probability. Bernoulli NB is better than Multinomial for short documents where presence/absence matters more than frequency. + +Naive Bayes is among the fastest classifiers to train and predict. Training is O(n × d) — a single pass over the data to compute priors and conditional probabilities. Prediction is O(d × K) — computing K class-conditional products for d features. No iterative optimization, no hyperparameters (beyond the smoothing parameter α). The model is naturally online — it can be updated incrementally as new data arrives. However, when features are highly correlated (which is common), the independence assumption leads to overly confident probabilities — the model pushes probabilities toward 0 or 1. Calibration techniques (Platt scaling, isotonic regression) can fix this when probability estimates matter. + +### Example + +**Spam detection (50,000 emails, 10,000 features after TF-IDF):** Prior P(spam) = 0.20, P(ham) = 0.80 from training data. For a new email containing "free", "money", "winner", and "meeting": P(email|spam) = P(free|spam) × P(money|spam) × P(winner|spam) × P(meeting|spam) = 0.15 × 0.08 × 0.12 × 0.02 = 2.88 × 10⁻⁵. P(email|ham) = 0.01 × 0.005 × 0.002 × 0.15 = 1.5 × 10⁻⁸. Posterior odds: P(spam|email) / P(ham|email) = 0.20 × 2.88 × 10⁻⁵ / (0.80 × 1.5 × 10⁻⁸) = 4800. P(spam|email) ≈ 4800/4801 = 99.98% — clear spam classification. With Laplace smoothing α=1, a word not seen in training (e.g., "bitcoin") doesn't zero out the probability — it gets a small non-zero probability of 1/(n_words + α × n_classes). Test set accuracy: 97.2%, AUC-ROC: 0.991, training time: 2.3 seconds. + +### Interview Questions (2-3) + +**Q: Why does Naive Bayes work well for text classification despite the independence assumption being obviously violated?** +A: In text, words are clearly correlated ("doctor" and "hospital" co-occur), violating conditional independence. Yet Naive Bayes works because: (1) The assumption only needs to hold approximately for the **ranking** of classes, not for exact probability estimation. As long as the relative magnitudes of P(spam|email) vs P(ham|email) are correct, classification is correct. (2) In high-dimensional sparse spaces like text, many correlations cancel out across the feature set. (3) The exponential family form concentrates probability mass correctly even when independence is violated. (4) The model's bias (high bias due to the wrong assumption) is often outweighed by extremely low variance — Naive Bayes has few parameters to estimate and requires less data to converge. Hand and Yu (2001) showed Naive Bayes is optimal in many settings where independence doesn't hold because the decision boundary remains the same. + +**Q: Compare Multinomial vs Bernoulli Naive Bayes for text classification.** +A: Multinomial NB models word **counts** or frequencies — it considers how many times a word appears. Bernoulli NB models binary word **presence** — whether a word appears or not. Multinomial is generally better for longer documents where word frequency matters (e.g., a spam email that mentions "free" 10 times is more suspicious than one mentioning it once). Bernoulli is better for short documents (tweets, search queries) where length is constrained and presence is more informative than frequency. In practice, try both: Multinomial typically wins for news articles and full emails, Bernoulli for SMS and social media posts. A key difference: in Multinomial NB, words that appear more often in ham than spam still reduce the spam probability multiplicatively; in Bernoulli NB, a word's absence from a document also carries information (P(word=0|class)). + +**Q: How do you handle zero probabilities in Naive Bayes?** +A: If a word never appears in the training data for a given class, its conditional probability P(word|class) = 0, which zeros out the entire product for that class. Laplace smoothing (add-α smoothing) fixes this by adding α to each count: P(word_i | class) = (count(word_i, class) + α) / (count_all_words(class) + α × V), where V is the vocabulary size. α=1 is Laplace smoothing; α < 1 is Lidstone smoothing. Higher α smooths more aggressively, pulling probabilities toward the uniform distribution. Cross-validation selects α: for Multinomial NB, α=0.1-0.5 often works better than α=1. Another approach: use log-probabilities instead of multiplying raw probabilities to avoid underflow in long documents. + +--- + +## Hierarchical Clustering +**Definition:** Builds a hierarchy of clusters via agglomerative (bottom-up merging) or divisive (top-down splitting) approaches. Produces a dendrogram for visual interpretation. No predetermined K required. + +### Theory & Explanation (2-4 paragraphs) + +Hierarchical clustering creates a tree-like structure (dendrogram) representing nested groupings of data points at various granularities. **Agglomerative clustering** (bottom-up) starts with each point as its own singleton cluster and iteratively merges the two closest clusters until one cluster remains. This is by far the more common approach due to its computational efficiency O(n³ naive, O(n² log n) with priority queues) and intuitive interpretation. **Divisive clustering** (top-down) starts with all points in one cluster and recursively splits, which is computationally expensive O(2^n) and rarely used. The result of either approach is a dendrogram — a tree diagram where the height of each merge represents the dissimilarity at which clusters were joined. + +The choice of **linkage criteria** determines which clusters merge at each step and significantly affects the resulting cluster shapes. **Single linkage** merges clusters based on the minimum distance between any point in cluster A and any point in cluster B. This can capture elongated, chain-like clusters but suffers from **chaining** — a single bridge point can cause unrelated groups to merge prematurely. **Complete linkage** merges based on the maximum pairwise distance, producing compact, spherical clusters but sensitive to outliers. **Average linkage** (UPGMA) merges based on the mean pairwise distance, balanced between single and complete. **Ward's method** merges clusters that minimize the increase in total within-cluster variance (sum of squared errors) — it tends to create equally-sized spherical clusters and is the most commonly used linkage for continuous numerical data, often producing results similar to K-Means but with a hierarchical view. + +The dendrogram is the key analytical output. By "cutting" the dendrogram at a given height (dissimilarity threshold), you obtain a flat clustering at the desired granularity. The elbow in the merge distance plot (similar to K-Means elbow) suggests natural cut points. Hierarchical clustering is deterministic (no random initialization), produces the same result each time on the same data, and doesn't require specifying K beforehand — you can choose the number of clusters after inspecting the dendrogram. However, it has O(n²) memory complexity (storing the full distance matrix), making it impractical for datasets larger than 10,000-20,000 points. It also cannot reassign points after merging (a mistake in an early merge propagates), unlike K-Means which iteratively reassigns. + +### Example + +**Customer segmentation on 500 mall customers (age, income, spending score):** Using Ward linkage on Euclidean distances, the dendrogram shows three distinct merges at height 150 — cutting at height 100 gives 4 clusters. Cluster 1 (n=180): high income, high spending ("Premium"). Cluster 2 (n=140): moderate income, low spending ("Budget-conscious"). Cluster 3 (n=110): young, low income, high spending ("Trendy youth"). Cluster 4 (n=70): older, high income, low spending ("Wealthy savers"). The height of the merge for clusters 3 and 4 is 85, while the height for joining clusters 1 and 2 with 3+4 is 145 — the dendrogram visually confirms that the trendy youth and wealthy savers are more similar to each other than to the other segments. Comparing with K-Means (K=4): silhouette score is 0.34 for hierarchical vs 0.31 for K-Means, confirming hierarchical better captures the natural group structure. + +### Interview Questions (2-3) + +**Q: When would you choose hierarchical clustering over K-Means?** +A: Choose hierarchical clustering when: (1) You don't know the number of clusters and want to explore the data structure visually via a dendrogram. (2) You suspect clusters have complex shapes or nested structures (e.g., sub-clusters within clusters). (3) You want a deterministic, reproducible result without random initialization. (4) Your dataset is small to moderate (n < 10,000) — hierarchical clustering is O(n²) in memory and O(n³) in time. (5) You need a hierarchical taxonomy (e.g., biological taxonomy, product categories). Choose K-Means when: (1) You have a large dataset (100K+ points) — K-Means is O(n) per iteration. (2) You know (or can estimate) K. (3) You need speed and scalability over hierarchical structure. (4) Data is high-dimensional — hierarchical clustering on Euclidean distances breaks down in high dimensions (curse of dimensionality). + +**Q: Compare single, complete, and Ward linkage. What type of clusters does each produce?** +A: Single linkage produces long, elongated, "stringy" clusters that can capture non-spherical shapes (e.g., a winding river of points). It's good for finding connected components but susceptible to chaining — a single noisy point can bridge two well-separated clusters. Complete linkage produces compact, tight, spherical clusters of roughly equal diameter. It's robust to chaining but sensitive to outliers — one outlier far from both groups can force two legitimate clusters to stay separate longer than they should. Ward linkage minimizes the increase in within-cluster variance, producing compact spherical clusters of roughly equal size. Ward is the preferred default for continuous data because its objective function (minimizing SSE) matches K-Means's objective, making the hierarchy consistent with a flat partition strategy. Imagine three clusters of points in 2D: two circular, one elongated. Single finds the elongated shape naturally; complete splits it into circles; Ward approximates the circles well but misses the elongated shape. + +**Q: How do you interpret a dendrogram and choose the optimal cut point?** +A: The dendrogram's vertical axis represents the dissimilarity (distance) at which clusters merge. Horizontal lines connect merged clusters. To determine a cut point: (1) Look for the largest vertical gap between consecutive merges — this suggests a natural clustering level. (2) Plot merge distance vs merge step — a sharp knee (elbow) suggests the optimal cut. (3) The inconsistency coefficient (comparing a merge's height to the average height of nearby merges) flags unusually large jumps. (4) Domain knowledge: cut at a height that produces interpretable clusters. For example, in a customer segmentation dendrogram, cutting at height 200 produces 2 clusters (high vs low spenders), cutting at 150 produces 4 clusters (more nuanced segments), and cutting at 50 produces 12 clusters (oversegmented). The "right" cut depends on the granularity needed for the business application. + +--- + +## DBSCAN +**Definition:** Density-based spatial clustering (Density-Based Spatial Clustering of Applications with Noise). Groups points that are closely packed together, marking points in low-density regions as noise. Does not require specifying the number of clusters. + +### Theory & Explanation (2-4 paragraphs) + +DBSCAN defines clusters as maximal sets of points connected by density-reachability. Two parameters control this: **epsilon (ε)** — the radius of the neighborhood around each point, and **minPts** — the minimum number of points required to form a dense region (including the point itself). A point is a **core point** if at least minPts points (including itself) are within distance ε. A **border point** is within ε of a core point but has fewer than minPts in its own ε-neighborhood. A **noise point** is neither a core nor border point — it lies in a low-density region. Clusters form around core points: any two core points within ε of each other are density-connected and belong to the same cluster; border points are assigned to the cluster of their nearest core point. + +The algorithm scans each point once. For each unvisited point, DBSCAN queries all points within ε distance. If the point is a core point, a new cluster is formed and expanded by recursively adding all density-reachable points (following the chain of core points). If the point is not a core point, it's temporarily marked as noise — it may later be reclassified as a border point if it falls within ε of a core point discovered later. The result is that DBSCAN naturally discovers arbitrarily shaped clusters (not just spherical), handles noise robustly, and requires no pre-specified number of clusters. Time complexity is O(n log n) with spatial indexing (R-tree, KD-tree) or O(n²) without. + +The key challenge is parameter selection. ε is data-dependent — too small fragments clusters into many tiny clusters with most points classified as noise; too large merges distinct clusters into one. The **k-distance graph** heuristic helps: for each point, compute the distance to its k-th nearest neighbor (k = minPts - 1), sort these distances, and look for the "elbow" — the distance where the curve steepens. That elbow is a reasonable ε estimate. minPts is domain-dependent but a rule of thumb is minPts ≥ D + 1 (where D is dimensionality), with 2×D being a safer lower bound. In 2D data, minPts=4 is common; in higher dimensions, minPts = 10-20. DBSCAN struggles with varying density clusters — if one cluster is dense and another is sparse, a single ε value cannot capture both. OPTICS is a generalization that addresses this by creating an augmented ordering representing density-based cluster structure across varying ε values. + +### Example + +**Geographic clustering of 2,000 retail store locations:** Using haversine distance (great-circle distance on Earth's surface), ε = 0.02 (≈ 2.2 km), minPts = 5. DBSCAN identifies 23 clusters (urban areas with dense retail concentration) and 187 noise points (rural standalone stores). Largest cluster: downtown metropolitan area with 312 stores within 2 km radius. The algorithm naturally captures irregular urban shapes — a retail corridor along a highway is a single elongated cluster rather than being split into multiple spherical clusters (as K-Means would do). In contrast, K-Means (K=23) splits the largest metropolitan cluster into 3 arbitrary pieces and assigns rural stores to the nearest urban center. Silhouette score: DBSCAN = 0.47 (only noise points have negative silhouettes), K-Means = 0.33. The noise detection is practically valuable — the 187 isolated stores can be flagged for different operational strategies. + +### Interview Questions (2-3) + +**Q: When would you choose DBSCAN over K-Means?** +A: Choose DBSCAN when: (1) Clusters have arbitrary shapes (crescents, S-curves, rings) — K-Means forces spherical clusters. (2) The data contains noise and outliers — DBSCAN explicitly labels noise points rather than forcing every point into a cluster. (3) You don't know the number of clusters — DBSCAN discovers K automatically. (4) Clusters vary in size — K-Means tends to create equal-sized clusters due to its objective function. Choose K-Means when: (1) Clusters are known to be spherical and roughly equal-sized (e.g., quantized sensor readings). (2) You have high-dimensional data — DBSCAN breaks down in high dimensions due to the concentration of distances (all points appear equally far apart). (3) You need a fast, scalable solution for large datasets. (4) Clusters have similar density throughout the space — DBSCAN's single ε cannot handle varying densities. + +**Q: Explain core points, border points, and noise points. What happens if ε is too small or too large?** +A: Core points have ≥ minPts neighbors within ε — they are the "dense interior" of clusters. Border points have < minPts neighbors but sit within ε of a core point — they are the "fringe" of clusters. Noise points have < minPts neighbors and are not within ε of any core point — they belong to no cluster. If ε is too small: too few points have ≥ minPts neighbors, creating many small fragment clusters; most points become noise or border points; the k-distance elbow is below the actual density threshold. If ε is too large: the neighborhood radius is too generous, merging distinct clusters; almost all points become core points; noise is eliminated but at the cost of oversmoothing cluster boundaries. The ideal ε is the k-distance elbow where the sorted distances to the k-th nearest neighbor sharply increase — this is the transition zone between dense cluster interiors and sparse between-cluster regions. + +**Q: What are DBSCAN's limitations and how does OPTICS address them?** +A: DBSCAN's primary limitation is its inability to handle clusters of varying density. With a single ε, either the dense cluster fragments (ε too small for sparse regions) or sparse clusters merge (ε too large for dense regions). Second, DBSCAN fails in high dimensions — the curse of dimensionality makes all pairwise distances similar, and density becomes meaningless. Third, DBSCAN's minPts and ε must be manually specified and are sensitive; small changes produce very different results. OPTICS (Ordering Points To Identify the Clustering Structure) addresses the varying density problem by computing the **reachability distance** for each point — the smallest ε that would make it density-reachable from a core point. Instead of producing a single flat clustering, OPTICS outputs a reachability plot (a 1D ordering of points with distance on the y-axis). Valleys in this plot correspond to clusters at different density levels — you can extract clusters at multiple density thresholds from a single OPTICS run. OPTICS is more robust but computationally more expensive (comparable to DBSCAN with indexing). + +--- + +## t-SNE +**Definition:** t-Distributed Stochastic Neighbor Embedding — non-linear dimensionality reduction technique for visualizing high-dimensional data in 2D or 3D. Preserves local neighborhood structure rather than global distances. + +### Theory & Explanation (2-4 paragraphs) + +t-SNE converts pairwise similarities between high-dimensional points into joint probabilities and then minimizes the Kullback-Leibler (KL) divergence between these probabilities in the original space and the embedding space. In the high-dimensional space, similarity between point i and j is modeled as a Gaussian centered at i: p_{j|i} = exp(-||x_i - x_j||² / 2σ_i²) / Σ_{k≠i} exp(-||x_i - x_k||² / 2σ_i²). The variance σ_i is chosen per point to achieve a user-specified **perplexity** (roughly, the effective number of neighbors each point considers). In the low-dimensional embedding, similarities are modeled using the Student-t distribution (with one degree of freedom): q_{ij} = (1 + ||y_i - y_j||²)^{-1} / Σ_{k≠l} (1 + ||y_k - y_l||²)^{-1}. The heavy-tailed t-distribution avoids the "crowding problem" — moderate distances in high-dimensional space are mapped to larger distances in 2D, creating better separation between clusters. + +The **perplexity** parameter is the most important tuning knob, typically set between 5 and 50. It represents a smooth measure of the effective number of neighbors balanced by each Gaussian kernel. Low perplexity (5-10) focuses on very local structure, often producing many tiny clusters; high perplexity (30-50) incorporates more global structure, producing larger, more spread-out clusters. t-SNE is remarkably effective at revealing clusters that exist in the original high-dimensional space — it often creates visually striking plots where distinct groups separate cleanly. However, the absence of visible clusters does not prove the data lacks structure; it may simply mean the data is better captured by a continuous manifold rather than discrete groupings. + +Critical caveats for t-SNE: (1) **Distance preservation is not global** — distances between clusters in the embedding are arbitrary; cluster sizes in the plot do not reflect anything meaningful. (2) **Stochastic** — different runs with the same data produce different visualizations (random seed and initialization matter). (3) **Perplexity selection** dramatically changes the plot — always explore multiple perplexity values. (4) **The cost function is non-convex** — the optimization can get stuck in local minima; running multiple times with different random seeds is recommended. (5) **Not for feature extraction or downstream modeling** — t-SNE embeddings are data-dependent and non-parametric, meaning they can't generalize to new points (though parametric t-SNE variants exist). UMAP is a modern alternative that preserves more global structure while being faster. + +### Example + +**Visualizing MNIST digits (70,000 images, 784 pixels each):** PCA projection explains only 20% of variance in the first 2 components — the 10 digits overlap substantially. t-SNE (perplexity=30, 1000 iterations, learning_rate=200) produces a 2D embedding where the 10 digit classes separate into distinct, well-separated clusters. Digits that are visually similar (3 vs 5, 4 vs 9) appear close to each other in the t-SNE map, while visually distinct digits (0 vs 8) are far apart. The 1's cluster is compact and well-separated (1's are distinctive), while the 8's and 9's clusters have fuzzy boundaries (these digits have high intra-class variation). Running t-SNE with perplexity=5 shows 80+ tiny clusters as the algorithm picks up fine-grained writing style differences. With perplexity=50, clusters are larger and more blended. Training time: approximately 5 minutes on a single CPU (O(n²) complexity). UMAP on the same data takes 30 seconds with comparable visual structure. + +### Interview Questions (2-3) + +**Q: Why does t-SNE use a t-distribution in the low-dimensional space but a Gaussian in the high-dimensional space?** +A: The Gaussian distribution in high-dimensional space models local neighborhoods appropriately — the probability mass drops exponentially with squared distance, effectively zeroing out distant points. In the low-dimensional embedding, using the same Gaussian would create the "crowding problem": high-dimensional space has vastly more volume in the periphery than 2D space, and moderate distances that should map to moderate distances in 2D would be "crowded" by many points at the same moderate distance. The t-distribution (heavy-tailed) assigns higher probability to moderate distances in low-dimensional space, effectively "stretching" the embedding and creating wider separations between clusters. Mathematically, this gradient formulation creates an attractive force that pulls similar points together and a repulsive force that pushes dissimilar points apart. The t-distribution ensures that the repulsive force from moderately similar points doesn't dominate, allowing clear cluster separation. + +**Q: What are the key limitations of t-SNE and when should you use UMAP instead?** +A: Key limitations: (1) Non-parametric — cannot embed new out-of-sample points without retraining. (2) Slow O(n²) — impractical for datasets > 100K points without approximations (Barnes-Hut reduces to O(n log n) but loses detail). (3) Destroys global structure — cluster sizes, distances between clusters, and global geometry are not preserved. (4) Sensitive to perplexity, learning rate, and random seed — always check multiple settings. (5) Sometimes creates "false patterns" — the KL divergence objective tends to draw well-separated clusters even when the data has a continuous structure (check with PCA first). Use UMAP when: (1) You need both local and global structure preservation — UMAP preserves more of the global manifold topology. (2) You need speed — UMAP is 5-10× faster than t-SNE. (3) You need to embed new data points — UMAP has a transform method. (4) You have > 100K points. Use t-SNE when: (1) You want the most widely-studied, extensively validated visualization method (thousands of papers use it). (2) Cluster separation is the primary goal (t-SNE often creates cleaner separation than UMAP). (3) You have under 50K points. + +**Q: How does perplexity affect the t-SNE embedding?** +A: Perplexity is the effective number of neighbors each point considers. Low perplexity (2-10) focuses heavily on local neighborhood preservation — each point considers only its nearest neighbors, creating many small, tight clusters. This can reveal fine-grained structure but also amplifies noise, potentially fragmenting meaningful clusters into spurious sub-clusters. High perplexity (30-100) incorporates more global perspective — each point considers a broader neighborhood, leading to larger, more cohesive clusters. This can merge distinct but nearby clusters and obscure fine structure. The original paper recommends perplexity between 5 and 50. A robust practice: run t-SNE at multiple perplexity values (10, 30, 50) and look for structure that persists across settings — persistent structure is more likely to be real. The optimal perplexity for n data points is roughly n^(1/2), but this is a starting point, not a rule. + +--- + +## Isolation Forest +**Definition:** Anomaly detection algorithm that isolates outliers via random recursive partitioning. Anomalies are few and different — they require fewer random splits to isolate than normal points. + +### Theory & Explanation (2-4 paragraphs) + +Isolation Forest takes a fundamentally different approach from typical distance-based or density-based outlier detection methods. Instead of modeling "normal" behavior and flagging deviations, it directly targets isolation — how easily a point can be separated from the rest. The algorithm builds an ensemble of random decision trees (isolation trees — iTrees). For each tree, it randomly samples a subset of data (typically 256 samples per tree), randomly selects a feature, and randomly chooses a split value between the feature's min and max. It recursively partitions the space until every point is isolated in its own leaf (or the tree reaches a height limit). The core insight: anomalous points are "few and different" — they fall into sparse regions of the feature space, so a random split has a high probability of isolating them early (near the root of the tree). Normal points are "many and similar" — they cluster in dense regions requiring many splits to separate. + +The detection score is based on the average path length across all trees in the forest. For each point x, compute h(x) = average depth (number of splits) across all iTrees. The anomaly score is: s(x, n) = 2^(-E[h(x)] / c(n)), where c(n) is the average path length for random data (theoretically derived as 2H(n-1) - 2(n-1)/n, where H is the harmonic number). Anomaly scores near 1 indicate clear outliers, scores near 0.5 suggest normal points, and scores consistently below 0.5 indicate highly concentrated normal points. The **contamination** parameter sets the expected proportion of outliers in the data — it's used for automatic thresholding: points with the highest anomaly scores (top contamination fraction) are flagged as outliers. + +Isolation Forest has several practical advantages. Training is O(n × t × log ψ) where n is data size, t is number of trees (typically 100), and ψ is the subsample size (256) — the algorithm doesn't need to see all data. It handles high-dimensional data well (no distance computations needed), requires no distance matrix O(n²) or density estimation, works without normalization (the random splits are threshold-based), and has low memory footprint. Key limitations: (1) It assumes anomalies are rare (contamination < 0.5). (2) It struggles when anomalies form localized clusters (many similar outliers). (3) It's sensitive to irrelevant features — adding noise dimensions dilutes the anomaly signal. (4) It doesn't produce interpretable explanations for why a point is anomalous (though feature importance can be extracted from split frequencies). + +### Example + +**Credit card fraud detection (284,807 transactions, 492 frauds = 0.17%):** Isolation Forest with 100 trees, max_samples=256, contamination=0.0017 (auto). Fraudulent transactions have average path length of 4.2 across trees, while legitimate transactions average 11.8. Anomaly score for a typical fraud: s = 2^(-4.2/10.5) = 0.87. For a typical legitimate: s = 2^(-11.8/10.5) = 0.46. At the default threshold (contamination=auto), recall = 0.73 (359 frauds caught out of 492), precision = 0.68 (unusual legitimate transactions flagged as outliers). False positive rate = 0.08% (about 227 false alarms from 284K legitimate). In comparison, a One-Class SVM on the same data takes 45 minutes to train (vs 12 seconds for Isolation Forest) and achieves recall = 0.69 with precision = 0.71. The Local Outlier Factor (LOF) achieves recall = 0.67 with precision = 0.65 in 8 minutes. Isolation Forest is 225× faster than One-Class SVM with comparable detection quality. + +### Interview Questions (2-3) + +**Q: Explain the intuition behind "isolating anomalies is easier than isolating normal points."** +A: Imagine a 2D scatter plot. Normal points cluster densely in a region — to isolate a single normal point, you need many random cuts to separate it from its neighbors (like repeatedly cutting a dense forest to isolate one tree). An anomaly sits alone in a sparse region — a single random cut is likely to separate it from everything else. Mathematically, the path length to isolate a point is roughly proportional to the density of its region. The Isolation Forest formalizes this: it measures the average number of random splits needed to isolate each point. Since anomalies are few and their feature values are extreme compared to normal points, they systematically require fewer splits. This intuition holds regardless of the data dimensionality — the probability that a random split separates an anomaly from the dense cluster depends only on the relative size of the anomaly's gap from the cluster, not on the dimension count. + +**Q: How do you choose the contamination parameter and what happens if it's wrong?** +A: Contamination is the expected proportion of outliers in the data. It's used to set the anomaly score threshold: the top contamination fraction of points by anomaly score are flagged. If contamination is set too low: many true anomalies are missed (high false negative rate) — points that should be flagged fall just below the cutoff. If contamination is too high: normal points are falsely flagged as anomalies (high false positive rate). In practice: (1) Use domain knowledge (e.g., in fraud detection, you know approximately 0.1-1% of transactions are fraud). (2) Set contamination="auto" — the threshold is set at the theoretical anomaly score cutoff for random data (0.5). (3) Use the contamination-free approach: sort all points by anomaly score, examine the top N candidates manually, and set a business-relevant threshold based on the cost of investigation vs the cost of missing a true anomaly. (4) For unlabeled data, use the elbow method on sorted anomaly scores. Isolation Forest's scores are relatively stable across runs, so the top-100 flagged points are consistently the most anomalous. + +**Q: What are Isolation Forest's limitations compared to distance-based methods like LOF?** +A: Isolation Forest assumes anomalies are "few and different" — rare points with extreme feature values. It struggles when: (1) Anomalies form local clusters (many similar outliers) — a cluster of 50 fraud accounts that look similar to each other but different from normal points will need more splits to isolate than a single fraud account, potentially being classified as normal. (2) Anomalies are "local" — a point that's normal relative to its local neighborhood but anomalous globally (e.g., a legitimate high-spending customer in a region of low spenders). LOF excels at this scenario because it compares local density. (3) Data has many irrelevant features — each noise feature adds random split points that dilute the anomaly signal. (4) The contamination ratio is high (> 0.3) — the assumption that anomalies are rare breaks down. Distance-based methods like LOF are better for local anomalies and handle varying-density regions well, but scale poorly (O(n²) neighborhood computations) and break down in high dimensions due to the distance concentration phenomenon. + +--- + +## Stacking +**Definition:** Ensemble meta-learning technique where multiple base models (level 0) are trained and their predictions are used as features to train a meta-model (level 1). Combines diverse model strengths for improved accuracy. + +### Theory & Explanation (2-4 paragraphs) + +Stacking (Stacked Generalization) trains a multi-layer ensemble where base models make predictions and a meta-model learns how to best combine them. In its simplest two-level form: **Level 0** consists of diverse base models trained on the original data (e.g., logistic regression, random forest, SVM, XGBoost). **Level 1** is a meta-model trained on the predictions (or predicted probabilities) of the base models as features, optionally including the original features. The meta-model learns which base models to trust in which regions of the feature space — it automatically identifies that SVM is more reliable for one subset of examples while Random Forest dominates another, rather than forcing a fixed model weighting. + +The critical technique is **k-fold cross-validation for generating meta-features**. If base models predict on the same data they were trained on, their predictions would be overfit and the meta-model would learn noise rather than genuine patterns. Instead, for each base model, perform k-fold cross-validation: train on k-1 folds, predict on the held-out fold, and repeat for all folds. The out-of-fold predictions for all training instances become the training features for the meta-model. After generating these meta-features, each base model is retrained on the full training set so it can make predictions on new data. This two-stage process ensures the meta-model sees honest (unbiased) predictions from the base models. Without this, stacking would severely overfit and likely underperform the best individual model. + +The choice of meta-model matters. Logistic regression is a common and effective choice — it learns weights for each base model's contribution, producing an interpretable ensemble that rarely overfits. A simple linear model on the meta-features essentially says: "Assign each base model a weight based on its reliability." More complex meta-models (Random Forest, XGBoost, neural networks) can capture interactions between base models (e.g., "When logistic regression and SVM agree, trust the prediction; when they disagree, defer to Random Forest"). However, complex meta-models risk overfitting the meta-features (especially if only a small number of training examples are available). A robust approach: use simple meta-models with few base models, then add base models and meta-model complexity as data increases. + +### Example + +**House price prediction competition (1,460 training examples):** Level 0: ElasticNet (regularized linear model), Random Forest (100 trees), XGBoost (n_estimators=500, learning_rate=0.05), and a Gradient Boosting Regressor. Using 5-fold cross-validation, each base model produces out-of-fold predictions. Meta-features: 4 columns (one per base model). Meta-model: Ridge regression. Performance (RMSE on test set): ElasticNet = 0.142, Random Forest = 0.138, XGBoost = 0.125, Gradient Boosting = 0.129, Stacking = 0.119. The stacking ensemble beats every individual model, achieving a 4.8% improvement over the best single model (XGBoost). Adding the original 80 features to the meta-model (in addition to the 4 predictions) further reduces RMSE to 0.117. The Ridge coefficients show: XGBoost gets weight 0.52, Gradient Boosting 0.28, Random Forest 0.15, ElasticNet 0.05 — the meta-model correctly learns that tree-based models are more predictive than the linear model for this problem. + +### Interview Questions (2-3) + +**Q: Explain the difference between stacking and simple averaging/voting ensembles.** +A: Simple averaging (regression) or majority voting (classification) weights each model equally or by a fixed weight. Stacking learns **optimal, potentially non-linear weighting** from the data. Simple averaging assumes all models are equally reliable everywhere. Stacking recognizes that one model may be better in some regions of the feature space while a different model dominates elsewhere. For example, in a binary classification problem: Logistic Regression might be more reliable for examples near the decision boundary, Random Forest might be better on examples with missing features, and SVM might excel on well-separated examples. Stacking's meta-model learns these regional preferences. The main downside is stacking adds complexity and overfitting risk — a simple average often works surprisingly well as a baseline, and the improvement from stacking is typically 1-3% over the best individual model when done properly. + +**Q: Why must meta-features be generated via cross-validation rather than using training predictions directly?** +A: If you train a base model on the training data and then predict on the **same training data**, the predictions are overfit — the model has already seen those examples. These overfit predictions convey a falsely optimistic picture of the model's ability. For instance, a severely overfit Random Forest might have near-perfect training predictions (R² = 0.99) but much worse test predictions. If the meta-model trains on these overfit predictions, it learns to heavily weight the Random Forest, expecting R² = 0.99 performance. On new data, the Random Forest performs far worse (R² = 0.70), and the stacking ensemble collapses. By using k-fold cross-validation to generate meta-features, the predictions for each training example come from a model that has never seen that example — they are "honest" estimates of generalization performance. This prevents information leakage from the base models to the meta-model and produces a stacking ensemble that generalizes properly. + +**Q: When would you avoid stacking despite its potential accuracy gains?** +A: Avoid stacking when: (1) **You need interpretability** — stacking adds a second opaque layer, making the final prediction harder to explain. Even with logistic regression as a meta-model, the combined decision boundary is complex. (2) **You have limited data** — with fewer than ~500 examples, the cross-validation step reduces training data to ~400 per fold, degrading base model quality, and the meta-model risks overfitting on the 4-10 meta-features. (3) **Your best single model already performs well** — the 1-2% improvement from stacking may not justify the increased complexity and maintenance burden. (4) **You need fast inference** — stacking requires running all base models plus the meta-model for each prediction, multiplying inference time by the number of base models. (5) **Base model predictions are highly correlated** — if all base models make the same mistakes, the meta-model has no signal to learn from; diversity among base models is essential for stacking to add value. A common mistake: stacking five models of the same type (e.g., five XGBoost variants) — the predictions are nearly identical, and stacking adds nothing over simple averaging. + +--- + +## Learning Curve +**Definition:** Plot of model performance (training and validation error) as a function of training set size. Diagnoses bias, variance, and whether collecting more data will improve performance. + +### Theory & Explanation (2-4 paragraphs) + +A learning curve plots training set size on the x-axis against the model's error (or accuracy) on both the training set and validation set. Typically, as training set size increases: (1) **Training error increases** — with few examples, the model can memorize almost perfectly; as more data arrives, it must generalize, making some training mistakes. (2) **Validation error decreases** — with few examples, the model overfits to noise; more data helps it learn genuine patterns. The two curves converge as data increases. The shape and gap between these curves reveals the model's bias-variance tradeoff and whether additional data will help. + +**High bias (underfitting) diagnosis:** Both training and validation error are high and converge quickly at a high error rate. Adding more data won't help — the curves have already plateaued. The model is too simple to capture the underlying pattern (or the patterns are genuinely weak). Fix: use a more complex model, add better features, reduce regularization, or engineer interaction terms. **High variance (overfitting) diagnosis:** Training error is very low (the model memorizes well) but validation error is significantly higher — a large gap between the curves. Adding more data will likely close this gap, reducing validation error further. The curves have not converged. Fix: collect more data, reduce model complexity (e.g., simpler trees, stronger regularization, fewer features), or use ensemble methods like bagging. + +**Practical interpretation:** Plot learning curves over a grid of training set sizes, typically from 10% to 100% of data. For each point, train the model on the sampled training set and evaluate on the full validation set (or a fixed holdout). Average over 5-10 random subsamples at each size to smooth the curve (the variance at small sample sizes can be large). Learning curves are also used to estimate the data requirements for a given performance target — by extrapolating the validation curve, you can estimate how much more data would be needed to reach, say, 95% accuracy. This is enormously valuable for budgeting data collection efforts. + +### Example + +**Customer churn prediction model (50K training examples):** Learning curve for a Random Forest classifier (max_depth=20, 100 trees). At 1,000 samples: training F1 = 0.92, validation F1 = 0.64 (large gap — severe overfitting with too few examples). At 10,000 samples: training F1 = 0.79, validation F1 = 0.72 (gap narrowing). At 50,000 samples: training F1 = 0.75, validation F1 = 0.73 (gap almost closed). Interpretation: Random Forest with max_depth=20 has relatively low variance and moderate bias. The training curve is still slightly above the validation curve, suggesting more data (100K+) might yield marginal improvement (validation F1 might reach ~0.74). The Random Forest approach seems appropriate — the limiting factor is model bias, not variance. In contrast, a logistic regression model on the same data: training F1 = 0.61, validation F1 = 0.60 at 50K samples — both curves converge quickly at a low value, indicating high bias and suggesting that adding more data won't help; a more expressive model is needed. + +### Interview Questions (2-3) + +**Q: How can learning curves diagnose whether collecting more data will help?** +A: Look at the gap between the training and validation curves and whether the validation curve has plateaued. If validation error is still decreasing at the largest training set size and the gap is still significant (validation error > training error), collecting more data will likely improve performance. If the validation curve has plateaued (flat) and the training curve is also flat near the same error level (small gap), collecting more data will not help — the model has reached its bias limit, and you need to increase model capacity or engineer better features. If the training curve is flat low and validation curve is flat high (large plateaued gap), the model has high variance that won't be resolved by more data alone — you need to reduce model complexity, add regularization, or improve data quality. The extrapolation of learning curves using inverse power law models (e.g., y = a × n^(-b) + c) can quantify the expected improvement from additional data, helping calculate the ROI of data collection efforts. + +**Q: Describe the typical shape of a learning curve for a model that is overfitting.** +A: For an overfitting model (high variance): at small training set sizes, training error is near zero (perfect memorization) while validation error is very high (poor generalization) — a massive gap. As training size increases, training error slowly rises (more data means harder to memorize everything) and validation error steadily decreases (more data helps the model find real patterns). The gap narrows but may not close completely within the available data. The training curve is below the validation curve across all training sizes. If you extrapolate the validation curve, it's still sloping downward — the gap would continue to narrow with more data, making further data collection beneficial. Classic example: a deep decision tree (no pruning) on a small dataset — training error stays at 0 until the dataset is too large to memorize, and validation error starts very high then drops. + +**Q: What does it mean if both training and validation curves converge at a high error?** +A: This signals **high bias (underfitting)** — the model is too simple to capture the data's structure. The model performs poorly on training data (it can't even memorize the training set well), and validation performance is similarly poor. Since both curves have already converged (little gap) and plateaued (no downward trend), adding more data will not improve performance — the model's capacity is the bottleneck. Solutions: (1) Use a more complex model (increase max_depth, increase n_estimators, use a non-linear model). (2) Engineer better features (polynomial features, interactions, domain-specific features). (3) Reduce regularization (decrease λ in Ridge, decrease min_samples_leaf in trees). (4) Try a fundamentally different algorithm family (e.g., switch from linear to tree-based or neural network models). (5) The data may genuinely have a low signal-to-noise ratio, in which case the model's performance ceiling is inherent to the problem. + +--- + +## Hyperparameter Tuning +**Definition:** Systematic search for the optimal set of hyperparameters that maximize model performance. Key methods: Grid Search, Random Search, Bayesian Optimization. + +### Theory & Explanation (2-4 paragraphs) + +Hyperparameters are model configuration parameters set before training begins (learning rate, tree depth, regularization strength, number of estimators) — distinct from model parameters learned during training (coefficients, tree splits, neural weights). The goal of hyperparameter tuning is to find the combination that minimizes generalization error. Every model has a hyperparameter space that grows combinatorially — if you have 5 parameters each with 10 possible values, the full Cartesian product is 100,000 combinations, many of which are impractical to evaluate. The three main approaches differ in how they navigate this space. + +**Grid Search** exhaustively evaluates all combinations in a predefined grid. It's simple, parallelizable, and guarantees finding the best combination within the grid — but it's computationally expensive and suffers from the curse of dimensionality. In high-dimensional hyperparameter spaces, grid resolution must be coarse to keep the number of evaluations tractable, meaning the true optimum may fall between grid points. **Random Search** (Bergstra & Bengio, 2012) samples hyperparameter combinations uniformly at random from the search space. The key insight: typically only 2-3 hyperparameters actually matter for performance, and random search explores more distinct values per important hyperparameter than grid search for the same budget. If you have a budget of 60 evaluations and 5 hyperparameters with 5 values each: grid search evaluates 5 values per hyperparameter; random search explores 60 values per hyperparameter (in expectation). Bergstra & Bengio showed random search finds better hyperparameters than grid search in 9 out of 10 tested cases — the paper title says it all: "Random Search for Hyper-Parameter Optimization." + +**Bayesian Optimization** builds a probabilistic model of the objective function (performance as a function of hyperparameters) and uses it to select the next promising combination to evaluate. It balances **exploration** (evaluating in uncertain regions) and **exploitation** (evaluating in regions known to perform well). Popular implementations: **Gaussian Processes** (scikit-optimize, GPyOpt) model the objective as a multivariate Gaussian with a kernel capturing similarity between hyperparameter combinations. **Tree-structured Parzen Estimators (TPE)** (Hyperopt, Optuna) model the density of good vs bad hyperparameter values using histograms or kernel density estimates — Optuna's TPE implementation is currently the most popular production choice. Bayesian Optimization typically outperforms random search for expensive evaluations (e.g., training a large neural network that takes hours per evaluation) but has overhead that may not be justified for fast training. + +### Example + +**Tuning XGBoost for a regression problem (5-fold cross-validation, 100K samples):** Manual baseline: max_depth=6, learning_rate=0.1, n_estimators=100, subsample=1.0, colsample_bytree=1.0. RMSE = 0.142. Grid Search (3×3×3 = 27 evaluations, ~10 minutes): max_depth ∈ [3, 6, 9], learning_rate ∈ [0.01, 0.1, 0.3], subsample ∈ [0.6, 0.8, 1.0]. Best: max_depth=6, learning_rate=0.1, subsample=0.8. RMSE = 0.138. Random Search (60 evaluations, ~22 minutes): samples from max_depth ∈ [3,15], learning_rate ∈ [0.001, 0.3] (log-uniform), subsample ∈ [0.5, 1.0], colsample_bytree ∈ [0.5, 1.0], reg_lambda ∈ [0, 10]. Best: max_depth=8, learning_rate=0.05, subsample=0.7, colsample_bytree=0.8, reg_lambda=1.5. RMSE = 0.131. Bayesian Optimization (Optuna, 100 trials, ~37 minutes): Automatically focuses on promising regions. Best: max_depth=7, learning_rate=0.03, n_estimators=850 (early stopping picks 520), subsample=0.75, colsample_bytree=0.85, reg_lambda=2.1, min_child_weight=5. RMSE = 0.128. The progression: Bayesian Optimization finds a 10% improvement over the baseline by exploring deeper trees with stronger regularization and lower learning rates. + +### Interview Questions (2-3) + +**Q: Explain why Random Search is often better than Grid Search for hyperparameter tuning.** +A: Bergstra & Bengio (2012) demonstrated that the function from hyperparameters to model performance is typically **low-effective-dimensionality** — not all hyperparameters matter equally. Usually 2-3 hyperparameters dominantly affect performance, while the rest contribute marginally. Grid Search samples each hyperparameter at a small fixed number of levels — if you have 5 parameters and budget for 50 evaluations, each parameter gets at most 3-4 values. Random Search samples each parameter independently from its full distribution — over 50 evaluations, each parameter is explored at 50 distinct values (in expectation). Since the important parameters could be any of the 5, random search has a much higher chance of finding good values for the truly important ones. Empirically, random search finds better configurations 9 out of 10 times for the same budget. Grid Search only wins when all parameters matter equally and the grid resolution is adequate, which is rare in practice. + +**Q: When would you use Bayesian Optimization instead of Random Search?** +A: Bayesian Optimization is worth the overhead when: (1) **Each evaluation is expensive** — training a deep neural network takes hours; spending a few extra seconds per iteration to decide the next hyperparameter is negligible overhead. (2) **You have a small to moderate evaluation budget** (< 100 evaluations) — Bayesian Optimization's smart sampling converges faster than random search. (3) **The search space is complex** — with categorical, integer, and conditional parameters (e.g., whether to use batch normalization, and if so, what momentum), Bayesian methods handle these naturally. Use Random Search when: (1) Training is fast (minutes, not hours) — the overhead of Bayesian Optimization isn't worth it. (2) You have abundant compute budget (1000+ evaluations) — random search with enough trials approaches full coverage. (3) You want maximum parallelization — random search is embarrassingly parallel while Bayesian Optimization is inherently sequential (each evaluation informs the next). (4) You need simplicity and reproducibility — Random Search with a fixed seed is trivial to implement and reason about. In practice, a common strategy: start with random search (50-100 trials) for initial exploration, then use Bayesian Optimization to fine-tune the most promising region. + +**Q: Describe a practical hyperparameter tuning strategy for a typical ML project.** +A: (1) **Start with defaults** — get a baseline using the model library's default parameters. This establishes a lower bound and validates the data pipeline. (2) **Identify critical hyperparameters** — for tree-based models: max_depth/num_leaves, learning_rate, n_estimators, min_samples_leaf, subsample. For neural networks: learning_rate, batch_size, number of layers/units, dropout, weight decay. For SVMs: C, gamma, kernel. (3) **Coarse-to-fine search** — first run random search (50-100 trials) with wide ranges covering orders of magnitude. Identify the promising region for each important parameter. (4) **Fine-tuning** — narrow the ranges around the best performing region from the coarse search. Run 50-100 more trials with Bayesian Optimization. (5) **Ensemble the winners** — train the final model using the top-5 hyperparameter configurations averaged together. (6) **Validate with cross-validation** — 5-fold CV for each evaluation; use the cv mean ± std to pick the winner, not just the mean (prefer configurations that perform well consistently over ones with high variance). (7) **Check sensitivity** — verify that small changes around the optimal hyperparameters don't cause large performance drops (stability matters for production). (8) For Kaggle competitions, add a "hyperparameter tuning" step after feature engineering is finalized, before model ensembling. + +--- + +## Log Loss +**Definition:** Logarithmic loss (cross-entropy loss) for probabilistic classification. Measures the difference between predicted probabilities and true labels. Lower is better, penalizes confident wrong predictions heavily. + +### Theory & Explanation (2-4 paragraphs) + +Log Loss (Logarithmic Loss), also known as **cross-entropy loss** or **logistic loss**, is the standard loss function for probabilistic classification. For binary classification: LogLoss = -(1/n) Σ [y_i × log(p_i) + (1 - y_i) × log(1 - p_i)], where y_i ∈ {0, 1} is the true label and p_i ∈ [0, 1] is the predicted probability that y_i = 1. For multi-class with K classes: LogLoss = -(1/n) Σ Σ y_ik × log(p_ik), where y_ik is a one-hot indicator and p_ik is the predicted probability for class k. The formula has a beautiful statistical interpretation: minimizing log loss is equivalent to maximizing the log-likelihood of the data under a Bernoulli (binary) or Multinomial (multi-class) distribution — it's the MLE estimator for classification. + +The behavior of log loss reveals why it's preferred over accuracy for probabilistic classifiers. A perfect prediction (p=1 for the true class) contributes 0 to the loss. A neutral prediction (p=0.5 for binary) contributes -log(0.5) = 0.693 per example. A confidently **wrong** prediction (p=0.001 when y=1) contributes -log(0.001) = 6.9 — over 10× the penalty of an unsure prediction. This asymmetric penalty structure is crucial: log loss forces the model to be not just correct but **well-calibrated** — the predicted probability should reflect genuine confidence. A model that predicts 90% confidence should be correct roughly 90% of the time. This is why log loss, not accuracy, is the proper metric for training probabilistic classifiers (though note: the model may be trained to minimize log loss directly). + +Log loss ranges from 0 (perfect) to infinity (worst). In practice: a good model achieves log loss of 0.1-0.3 for well-separable classes, 0.3-0.7 for moderate difficulty, and > 1.0 for difficult or noisy problems. A model that predicts p=0.5 for everything on balanced data achieves log loss = 0.693. A random model on imbalanced data (e.g., 90:10 always predicting the majority class at 90%) achieves log loss = -0.9 × log(0.9) - 0.1 × log(0.1) ≈ 0.325 — misleadingly low because the confidence matches the base rate. **Null model log loss** (always predicting the class prior) is a baseline — any useful model should beat it. The multi-class extension generalizes naturally, with a null model achieving log loss = -log(1/K) = log(K). + +### Example + +**Comparing log loss across three models for binary classification (10K test examples, 25% positive rate):** Model A (Logistic Regression): average p = 0.23 for negatives, 0.72 for positives. LogLoss = -[0.25 × log(0.72) + 0.75 × log(0.28)] ≈ 0.49 (taking the correct per-class averages). Model B (Random Forest): average p = 0.20 for negatives, 0.81 for positives. LogLoss = -[0.25 × log(0.81) + 0.75 × log(0.19)] ≈ 0.38. Model C (XGBoost with calibration): average p = 0.24 for negatives, 0.76 for positives. LogLoss = -[0.25 × log(0.76) + 0.75 × log(0.24)] ≈ 0.42. However, Model B's log loss is actually worse than 0.38 when computed per-example rather than averaged — some confident wrong predictions (p=0.95 for a negative) contribute -log(0.05)=3.0 to the sum, revealing miscalibration despite good average separation. After Platt scaling calibration, Model B's log loss improves to 0.36. The null model (always predicting p=0.25) achieves log loss = -log(0.25) = 1.39 — all three models are far better than random. Winner: Model B after calibration. + +### Interview Questions (2-3) + +**Q: Why does log loss penalize confident wrong predictions so heavily?** +A: The log function transforms small probabilities into large negative values: log(0.5) = -0.69, log(0.1) = -2.30, log(0.01) = -4.61, log(0.001) = -6.91. When a model predicts p=0.001 for a true positive, the contribution is -log(0.001) = 6.91 — roughly 10× worse than predicting p=0.50. This is deliberate: log loss is a proper scoring rule, meaning the optimal strategy is to predict the true conditional probability. If you predict overconfident probabilities, the penalty for being wrong is catastrophic enough to discourage this behavior. The asymmetry also correctly reflects real-world costs: in many applications, a confident wrong prediction is far more dangerous than an uncertain one. In medical diagnosis, predicting 99.9% "no cancer" when the patient actually has cancer is worse than predicting 55% "no cancer." A model cannot hide from its mistakes by being overconfident — log loss forces calibration. + +**Q: Compare log loss with AUC-ROC and accuracy for evaluating classifiers.** +A: Log loss measures **probability quality** — how close predicted probabilities are to the true probabilities. AUC-ROC measures **rank-ordering** — how well the model separates positive and negative scores, independent of calibration. Accuracy measures **hard classification** — whether the predicted class (at a given threshold) matches the true class, with no probability consideration. Log loss is the strictest metric: a model can have high AUC (excellent rank-ordering) but poor log loss (terrible calibration). Example: Model A predicts p=0.55 for all positives and p=0.45 for all negatives — AUC = 0.55 (barely better than random), log loss ≈ 0.69. Model B predicts p=0.99 for half of positives and p=0.50 for the other half — AUC could be 0.75 (good separation) but log loss could be 0.80 due to the confident mistakes on the poorly predicted positives. Use log loss when calibrated probabilities matter (risk modeling, medical diagnosis, bidding). Use AUC for model comparison and selection (it's threshold-independent). Use accuracy for the final business evaluation at a specific operating point. + +**Q: What does the log loss of the null model represent, and how do you use it?** +A: The null model predicts the training class prior for every example (e.g., p=0.25 for a 25% positive rate). Its log loss is -[prior × log(prior) + (1-prior) × log(1-prior)]. This provides a crucial baseline: any useful model must achieve log loss below the null model's. For balanced data (50% positive), null log loss = 0.693. For imbalanced data (1% positive), null log loss = -0.01 × log(0.01) - 0.99 × log(0.99) ≈ 0.056 — a very low baseline because the majority class dominates. A model achieving 0.040 log loss on 1% data may seem excellent, but relative to the null model, the improvement is only 28%. The **pseudo R²**, analogous to R² in linear regression, measures this improvement: pseudo R² = 1 - (model_logloss / null_logloss). A pseudo R² of 0.20 means the model explains 20% of the "uncertainty" beyond the null model. This is especially useful for comparing models across datasets with different class distributions — it normalizes for the base rate difficulty. diff --git a/techy/03-deep-learning-and-nlp.md b/techy/03-deep-learning-and-nlp.md new file mode 100644 index 0000000..3e41c2a --- /dev/null +++ b/techy/03-deep-learning-and-nlp.md @@ -0,0 +1,1102 @@ +# 3. DEEP LEARNING + +## Neural Networks (Neuron, Layer, Feedforward) +**Definition:** Computing systems inspired by biological neural networks — interconnected layers of neurons with weighted connections and non-linear activations learning complex patterns. + +### Theory & Explanation (3-5 paragraphs) + +A neuron, also called a perceptron, is the fundamental building block of a neural network. It takes multiple input values, multiplies each input by a corresponding weight, sums them up, adds a bias term, and then passes the result through a non-linear activation function. Mathematically, this is expressed as output = activation(W·X + b), where W represents the weight matrix, X is the input vector, and b is the bias. Each neuron essentially learns to fire or not fire based on patterns it detects in the data. The weights and biases are the learnable parameters that get adjusted during training. + +Neural networks organize neurons into layers: an input layer that receives the raw data, one or more hidden layers that perform intermediate computations, and an output layer that produces the final prediction. A feedforward network (also called a multilayer perceptron or MLP) passes data strictly forward: inputs go into the first hidden layer, activations flow to the next layer, and so on until the output layer produces a result. There are no cycles or feedback loops — hence the name feedforward. Each neuron in one layer connects to every neuron in the next layer (fully connected or dense), making these networks highly expressive but also parameter-heavy. + +The power of deep neural networks lies in hierarchical feature learning. Early layers learn simple patterns like edges in images or n-gram patterns in text, middle layers combine these into higher-level features like shapes or phrases, and deeper layers learn abstract concepts like objects or sentiments. The Universal Approximation Theorem states that a feedforward network with a single hidden layer containing a finite number of neurons can approximate any continuous function to arbitrary accuracy, provided the activation function is non-linear. However, the theorem does not say how many neurons are needed or whether the network can be trained effectively — deep networks with many layers often achieve better generalization with far fewer total parameters than a single massive hidden layer. + +Depth versus width is a fundamental tradeoff in neural network design. Wider networks (more neurons per layer) can memorize more patterns but risk overfitting and become computationally expensive. Deeper networks (more layers) learn hierarchical abstractions more efficiently and generalize better, but they suffer from vanishing gradients, optimization difficulties, and require careful initialization and regularization techniques. In practice, most modern architectures favor depth over width — ResNet has 152 layers, Transformers have 12-96 layers — and rely on skip connections, batch normalization, and sophisticated optimizers to train effectively. + +The learning process involves initializing weights (usually randomly with Xavier or He initialization), performing a forward pass to compute predictions, calculating a loss function that measures prediction error, and then using backpropagation to compute gradients and update weights via gradient descent. The network iterates through this cycle thousands or millions of times across the training data. Hyperparameters like learning rate, batch size, number of layers, and neurons per layer significantly impact training dynamics and final performance. Modern frameworks like PyTorch and TensorFlow automate gradient computation and provide building blocks, but understanding the underlying mechanics is essential for debugging and architecture design. + +### Example + +Consider a 3-layer feedforward network for predicting house prices. The input layer has 10 features (square footage, number of bedrooms, location scores, age, etc.). The first hidden layer has 64 neurons with ReLU activation, the second hidden layer has 32 neurons with ReLU activation, and the output layer has 1 neuron with linear activation (since price is a continuous value). The forward pass computation is: h1 = ReLU(W1·X + b1) where W1 is 64×10 and b1 is 64, h2 = ReLU(W2·h1 + b2) where W2 is 32×64 and b2 is 32, and finally y_pred = W3·h2 + b3 where W3 is 1×32 and b3 is a scalar. The number of parameters in the first hidden layer alone is 64 × 10 weights + 64 biases = 704 parameters. The total network has (640+64) + (2048+32) + (32+1) = 2,817 parameters total. For a batch of 32 houses, all computations are performed as matrix operations, making them highly efficient on GPUs. + +### Interview Questions (2-3) + +**Q: What is the Universal Approximation Theorem and why is depth still needed despite it?** +A: The Universal Approximation Theorem states that a single hidden layer feedforward network with a finite number of neurons can approximate any continuous function to arbitrary precision, given a non-linear activation function. However, depth remains important for three reasons. First, the theorem does not guarantee learnability — it only proves existence of such a network, not that gradient descent can find it. Second, achieving the same approximation with a shallow network may require exponentially more neurons than a deep network, making it computationally infeasible. Third, deep networks learn hierarchical representations naturally — early layers capture simple patterns, later layers build on them — which aligns with how complex real-world functions are structured. Deep networks generalize better from limited data because the hierarchical prior matches the compositional nature of many problems. + +**Q: How do you decide the number of layers and neurons for a neural network?** +A: There is no single formula, but several guidelines help. Start with a simple architecture (1-2 hidden layers) and gradually increase complexity while monitoring validation performance. For the number of neurons, a common starting point is to use powers of 2 (32, 64, 128, 256, 512) for computational efficiency on GPUs. The first hidden layer typically has more neurons than the input dimension to capture enough feature combinations, and subsequent layers gradually reduce dimension to compress information. Key heuristics include: (1) the number of parameters should be less than the number of training examples to avoid overfitting, (2) use dropout or early stopping as regularization rather than forcing fewer parameters, (3) start with a model larger than needed and regularize — it is easier to regularize an overfitting model than to fix an underfitting one, and (4) cross-validate or use validation curves to detect overfitting versus underfitting empirically. + +### Related Concepts +Activation Functions, Backpropagation + +## Activation Functions +**Definition:** Introduce non-linearity into neural networks — without them, stacked linear layers would collapse into a single linear transformation regardless of depth. + +### Theory & Explanation + +Activation functions are the crucial non-linear component that enables neural networks to learn complex patterns. Without activation functions, each layer would perform a linear transformation, and the composition of multiple linear transformations is itself a linear transformation. Mathematically, if every layer is just W·X + b, then two layers collapse to W2·(W1·X + b1) + b2 = (W2·W1)·X + (W2·b1 + b2), which is still a linear function. Non-linear activations break this equivalence, allowing the network to represent arbitrarily complex decision boundaries and functions. The choice of activation function significantly affects training dynamics, convergence speed, and final model quality. + +ReLU (Rectified Linear Unit), defined as f(x) = max(0, x), is the default activation for hidden layers in most modern architectures. It has several advantages: it avoids the vanishing gradient problem for positive inputs since its gradient is constant (1 for x > 0), it is computationally very cheap (just a max operation), and it induces sparsity since roughly half of the neurons output zero. However, ReLU suffers from the "dying ReLU" problem — when a neuron's weighted sum is always negative, its gradient is zero, and it can never recover, effectively becoming permanently dead. Variants like Leaky ReLU (f(x) = max(αx, x) with small α like 0.01) and Parametric ReLU (learnable α) address this by allowing a small gradient for negative inputs. + +Sigmoid activation, defined as σ(x) = 1/(1+e^(-x)), squashes any input to the range (0, 1), making it useful for binary classification probability outputs. However, it has significant drawbacks for hidden layers: it saturates for large positive or negative values (gradient approaches 0), causing vanishing gradients in deep networks; it is not zero-centered (outputs always positive), which can cause zigzagging gradients; and it involves computationally expensive exponential operations. For these reasons, sigmoid is almost never used in hidden layers of modern deep networks, though it remains common for the output layer of binary classifiers. + +Tanh (hyperbolic tangent), defined as tanh(x) = (e^x - e^(-x))/(e^x + e^(-x)), squashes inputs to the range (-1, 1) and is zero-centered, which partially mitigates the optimization issues of sigmoid. This zero-centered property means gradients can flow in both positive and negative directions, leading to smoother optimization. However, tanh still saturates for extreme inputs and has the vanishing gradient problem. It was historically popular for hidden layers before ReLU became dominant and is still used in some RNN variants and where bounded outputs are desired. + +Softmax is a multi-dimensional generalization of sigmoid used for multi-class classification. Given a vector of raw scores (logits) z = [z1, z2, ..., zK], softmax computes the probability of each class as p_i = e^(z_i)/Σ(e^(z_j)) for j=1 to K. The result is a probability distribution that sums to 1, with higher raw scores corresponding to higher probabilities. The exponential amplifies differences between scores, making the distribution sharper. Softmax is always used in the output layer for multi-class problems (with cross-entropy loss) and never as a hidden activation. The GELU (Gaussian Error Linear Unit) activation, defined as GELU(x) = x·Φ(x) where Φ is the standard Gaussian CDF, has become popular in Transformer architectures (BERT, GPT). It combines properties of ReLU, dropout, and stochastic regularization, providing a smooth non-linearity that outperforms ReLU in many NLP tasks. + +### Example + +Consider activation values for z = 3.2 and z = -3.2. For z = 3.2: ReLU outputs 3.2 (positive input passes through unchanged), Sigmoid outputs 1/(1+e^(-3.2)) = 0.96 (strong activation, almost saturated at 1), Tanh outputs tanh(3.2) ≈ 0.997 (strongly positive, near saturation). For z = -3.2: ReLU outputs 0 (dead neuron — no gradient flows back), Sigmoid outputs 1/(1+e^(3.2)) = 0.04 (weak activation, saturated near 0), Tanh outputs tanh(-3.2) ≈ -0.997 (strongly negative, saturated). This illustrates why ReLU is preferred — for positive inputs it has no saturation and full gradient flow, but negative inputs cause permanent death. In a batch of 128 examples fed through a 1024-neuron layer with ReLU, roughly half the neurons will be active, creating useful sparsity while maintaining gradient flow for the active ones. + +### Interview Questions + +**Q: Why is ReLU preferred over Sigmoid for hidden layers in deep networks?** +A: ReLU addresses three critical problems with Sigmoid in deep networks. First, the vanishing gradient problem — Sigmoid gradients saturate near zero for large positive or negative values, so in deep networks the gradient shrinks exponentially as it propagates backward, making early layers nearly untrainable. ReLU has a constant gradient of 1 for positive inputs, allowing gradients to flow freely through active neurons regardless of network depth. Second, ReLU is computationally much cheaper — it is a simple max(0, x) operation versus Sigmoid's exponential division, which matters when training models with billions of parameters across thousands of GPUs. Third, ReLU induces natural sparsity — approximately 50% of neurons output zero, which reduces computational cost and can improve generalization by creating distributed representations. The one downside is the dying ReLU problem, which can be mitigated with variants like Leaky ReLU or proper learning rate tuning. + +**Q: What is the dying ReLU problem and what are the fixes?** +A: The dying ReLU problem occurs when a neuron's weighted sum is always negative, causing ReLU to output 0 for every input in the training set. Since the gradient of ReLU at 0 is 0, the neuron can never update its weights to escape this state — it is permanently dead, contributing nothing to the network's capacity. This is particularly common with high learning rates that can push weights into regions where many neurons are always negative. Several fixes exist: (1) Leaky ReLU (f(x) = max(αx, x) with small α = 0.01) allows a small gradient for negative values, letting neurons potentially recover; (2) Parametric ReLU (PReLU) makes α a learnable parameter; (3) ELU (Exponential Linear Unit) uses f(x) = x for x > 0 and f(x) = α(e^x - 1) for x ≤ 0, providing smooth negative values; (4) proper weight initialization (He initialization specifically designed for ReLU) and careful learning rate scheduling; (5) batch normalization to keep activations centered and scaled, reducing the chance of systematic negative outputs. + +### Related Concepts +Neural Networks, Vanishing Gradient, ReLU + +## Backpropagation +**Definition:** The algorithm that computes the gradient of the loss function with respect to every weight in the network by applying the chain rule of calculus backward from the output to the input. It is the core training mechanism for all neural networks. + +### Theory & Explanation + +Backpropagation, often abbreviated as backprop, is the cornerstone of neural network training. It works in two phases. First, the forward pass takes an input, propagates it through each layer using the current weights and biases, applies activation functions, and produces a prediction. The loss function measures the error between this prediction and the true target value. Then, the backward pass computes how much each weight contributed to this error by applying the chain rule of calculus from the output layer back to the input layer. Specifically, for each layer, the algorithm computes the gradient of the loss with respect to that layer's inputs (δ), then uses δ to compute the gradient with respect to the layer's weights and biases. This is repeated layer by layer in reverse order. + +The chain rule makes backpropagation efficient and elegant. For a simple path loss → f → g → h → weight w, the gradient ∂loss/∂w = (∂loss/∂f)·(∂f/∂g)·(∂g/∂h)·(∂h/∂w). The key insight is that once we compute ∂loss/∂f at the output, we can reuse it to compute gradients for all preceding weights — we do not need to recompute from scratch for each weight. This gives backpropagation O(n) time complexity for n weights, making it feasible to train networks with millions or even billions of parameters. Without this efficiency, computing each weight's gradient independently would be O(n²), which would be infeasible for modern architectures. + +Vanishing and exploding gradients are fundamental challenges that backpropagation faces in deep networks. As gradients are multiplied layer by layer (by the chain rule), they can either shrink exponentially toward zero (vanishing) or grow exponentially (exploding). Vanishing gradients mean early layers learn extremely slowly or not at all, effectively limiting network depth. Exploding gradients cause numerical instability and training divergence. These problems are addressed through techniques including: careful weight initialization (Xavier for tanh/sigmoid, He for ReLU), batch normalization (normalizing layer outputs), gradient clipping (scaling down gradients that exceed a threshold), skip connections (allowing gradients to bypass layers, as in ResNet), and activation functions that preserve gradient magnitude (ReLU's constant 1 for positive inputs). + +Weight initialization is critical for effective backpropagation. If weights are initialized too large, activations grow and saturate, causing vanishing gradients. If too small, signals vanish before reaching deep layers. Xavier/Glorot initialization sets weights from a distribution with variance 2/(n_in + n_out), designed to keep the variance of activations constant across layers when using tanh or sigmoid. He initialization uses variance 2/n_in, specifically designed for ReLU activations which have different gradient flow properties. Proper initialization ensures that forward signals and backward gradients maintain reasonable scale throughout the network, allowing all layers to learn effectively from the start. + +Modern deep learning frameworks like PyTorch and TensorFlow automate backpropagation through automatic differentiation (autograd). They build a computational graph during the forward pass, recording every operation performed. During the backward pass, they traverse this graph in reverse, applying the chain rule at each node. This means practitioners rarely implement backpropagation manually, but understanding its mechanics remains essential for diagnosing training problems, implementing custom layers, and optimizing performance. Key practical considerations include: gradient accumulation over mini-batches, gradient checkpointing (trading compute for memory by recomputing activations), and mixed precision training (using 16-bit gradients to reduce memory and accelerate computation). + +### Example + +Consider a minimal 2-layer network for illustration. The input is a scalar x, the hidden layer uses ReLU, and the output is a scalar with mean squared error loss. Forward pass: z1 = w1·x + b1, a1 = ReLU(z1), y_pred = w2·a1 + b2, loss = (y_true - y_pred)². For the backward pass, we compute gradients from output to input. First, ∂loss/∂y_pred = 2(y_pred - y_true). Next, ∂loss/∂w2 = ∂loss/∂y_pred · ∂y_pred/∂w2 = (2(y_pred - y_true)) · a1. Similarly, ∂loss/∂b2 = 2(y_pred - y_true) · 1. For the hidden layer, ∂loss/∂a1 = ∂loss/∂y_pred · w2. Then ∂loss/∂z1 = ∂loss/∂a1 · ∂a1/∂z1 = ∂loss/∂a1 · (1 if z1 > 0 else 0) — note the ReLU derivative kills gradient for negative z1. Finally, ∂loss/∂w1 = ∂loss/∂z1 · x and ∂loss/∂b1 = ∂loss/∂z1. With a learning rate α, we update: w1 = w1 - α·∂loss/∂w1, and similarly for all parameters. This completes one training step. + +### Interview Questions + +**Q: Explain backpropagation in simple terms.** +A: Backpropagation is like a blame assignment algorithm for neural networks. Imagine a team of people (layers) passing a message forward to produce an answer. If the answer is wrong, you need to figure out how much each person contributed to the error. This is done by working backward from the final output: you compute how much changing the last person's work would change the error, then how much changing the second-to-last person's work affects the last person, and so on all the way back to the input. The chain rule from calculus provides the mathematical tool for this — it tells us exactly how sensitive each step is to the previous step. By multiplying these sensitivities along the chain, we get the gradient for every weight, which tells us the direction and magnitude to adjust each weight to reduce the error. The beauty is that this can be done efficiently: after one forward pass, the backward pass takes only about twice the computation and gives gradients for all weights simultaneously. + +**Q: What happens when gradients vanish and how do you detect and fix it?** +A: When gradients vanish, they shrink exponentially as they propagate backward through the network, becoming essentially zero in early layers. This means early layers receive negligible weight updates and fail to learn meaningful features. Signs include: (1) the loss stops decreasing after initial improvement, (2) earlier layer weights barely change between iterations, while later layer weights change significantly, (3) gradient norms drop sharply from the output to input layers (you can monitor gradient norms per layer during training). Common fixes include: (1) using ReLU or its variants instead of sigmoid/tanh since ReLU maintains a constant gradient of 1 for positive inputs; (2) batch normalization which prevents activations from saturating by normalizing each layer's inputs; (3) skip connections as in ResNet that provide gradient shortcuts bypassing layers; (4) proper weight initialization like He initialization for ReLU networks; (5) using optimizers like Adam that maintain per-parameter adaptive learning rates; (6) reducing network depth or using architectures specifically designed for very deep networks. + +### Related Concepts +Gradient Descent, Vanishing Gradient + +## CNN (Convolutional Neural Network) +**Definition:** A specialized neural network architecture designed for grid-like data, particularly images, that uses convolution operations and pooling layers to learn translation-invariant hierarchical features with far fewer parameters than fully connected networks. + +### Theory & Explanation + +Convolutional Neural Networks (CNNs) revolutionized computer vision by introducing the core principles of local connectivity and weight sharing. In a fully connected layer, each neuron connects to every input pixel — for a 224×224×3 image, that is over 150,000 weights per neuron. Convolutional layers instead use small filters (kernels), typically 3×3 or 5×5, that slide across the input image. Each filter connects only to a small local region called the receptive field. Crucially, the same filter weights are shared across all spatial positions — a filter that detects horizontal edges at the top of the image uses the same weights at the bottom. This weight sharing reduces parameters by orders of magnitude and provides translation invariance: if an object moves in the image, the same filter will detect it regardless of position. + +Local connectivity exploits the spatial structure of images: nearby pixels are highly correlated and form meaningful features (edges, textures), while distant pixels have weaker relationships. A 3×3 filter processes a local patch of 9 pixels, learning to detect local patterns like edges in specific orientations. Multiple filters (typically 32 to 512) are used in each layer, each learning to detect a different pattern. The output of applying all filters is a set of feature maps, where each feature map highlights where a particular pattern appears in the input. Stacking convolutional layers creates a feature hierarchy: early layers detect simple edges and color blobs, middle layers detect textures and shapes (eyes, wheels, windows), and deep layers detect complete objects (faces, cars, buildings). + +Pooling layers downsample feature maps, reducing spatial dimensions while retaining important information. Max pooling (the most common) takes the maximum value in each 2×2 or 3×3 window, discarding the rest. This provides several benefits: it reduces the number of parameters and computation for subsequent layers, it provides local translation invariance (a feature detected within the pooling window is preserved even if it shifts slightly), and it increases the receptive field of deeper neurons exponentially. The typical pattern is: Conv → ReLU → Pool, repeated several times, with the number of filters increasing as spatial dimensions decrease. This reflects the design principle of transitioning from low-level details to high-level abstractions. + +Key CNN architectures mark milestones in the field. LeNet-5 (1998) introduced the CNN paradigm for handwritten digit recognition. AlexNet (2012) won ImageNet with 8 layers, ReLU activations, and dropout, kicking off the deep learning revolution. VGGNet (2014) showed that stacking many 3×3 convolutions (16-19 layers) works better than larger filters. ResNet (2015) introduced skip connections enabling 152-layer networks and introduced the bottleneck block design. Inception (GoogLeNet) used parallel convolutions of different sizes (1×1, 3×3, 5×5) in the same layer. Each architecture represents a different design philosophy — deeper, wider, more parallel, or with better gradient flow — but all share the fundamental convolution + pooling + non-linearity building blocks. + +In recent years, Vision Transformers (ViTs) have challenged CNNs by applying Transformer self-attention directly to image patches. ViTs split images into 16×16 patches, treat each patch like a word token, and process them through Transformer layers. While ViTs can outperform CNNs with sufficient data, CNNs remain more data-efficient and are still widely used, especially when training data is limited. Modern architectures often combine both approaches (ConvNeXt, MaxViT). For deployment in production, CNNs typically have better latency and require less specialized hardware optimization than Transformers. At the EY L1 level, interviewers expect solid understanding of the core CNN concepts rather than the latest architectures. + +### Example + +Consider a CNN for classifying handwritten digits from the MNIST dataset (28×28 grayscale images). The architecture: Conv1 has 32 filters of size 3×3, producing 32 feature maps of size 26×26 (valid padding). ReLU activation follows. MaxPool1 with 2×2 window reduces to 13×13. Conv2 has 64 filters of size 3×3, producing 64 feature maps of size 11×11. ReLU, then MaxPool2 reduces to 5×5. At this point, the tensor has shape 64×5×5, which is flattened to 1600 features. A dense layer of 128 neurons with ReLU processes these features. Finally, a 10-neuron output layer with softmax produces probabilities for digits 0-9. Total parameters: Conv1 (32×3×3×1 + 32 biases = 320), Conv2 (64×3×3×32 + 64 = 18,496), Dense (1600×128 + 128 = 204,928), Output (128×10 + 10 = 1,290) = ~225,000 parameters. A fully connected network with the same input would require (28×28)×10 + 10 = 7,850 parameters for a single-layer classifier (too simple) or millions for a multi-layer network — the CNN achieves much better accuracy with orders of magnitude fewer parameters through weight sharing. + +### Interview Questions + +**Q: What is the fundamental difference between convolutional and fully connected layers?** +A: Three key differences. First, local connectivity: each neuron in a convolutional layer connects only to a small spatial region of the input (the receptive field, typically 3×3 or 5×5), while fully connected neurons connect to every input element. This matches the spatial structure of images where nearby pixels are highly correlated. Second, weight sharing: the same filter weights are applied across all spatial positions in the input, meaning a filter that detects edges learns one set of weights applied everywhere. Fully connected layers have separate weights for every connection. This weight sharing reduces parameters by orders of magnitude — a 3×3 filter has 9 weights regardless of input size, while a fully connected layer on a 224×224 image would have over 50,000 weights per output neuron. Third, translation invariance: weight sharing combined with pooling means CNNs can recognize patterns regardless of where they appear in the image — an edge detected at the top uses the same weights as one at the bottom. Fully connected layers must learn separate weights for each position and cannot generalize across spatial locations. + +**Q: What is a receptive field in a CNN and why does it matter?** +A: The receptive field is the region of the input image that influences a particular neuron. For a single 3×3 convolution on 28×28 input, each output neuron sees a 3×3 patch — a receptive field of 3×3. After two 3×3 convolutions, each neuron in the second layer sees a 5×5 region of the original image (each of the 3×3 positions in the second convolution sees a 3×3 patch with 1-pixel overlap, giving 5×5 effective coverage). After adding a 2×2 pooling layer, the receptive field expands further. The receptive field grows with depth: a VGG network with 13 convolutional layers (all 3×3) has an effective receptive field of 33×33 on the original image, while ResNet-152 covers a much larger region due to its depth and larger late-layer feature maps. The receptive field matters because it determines what context a neuron has for its decision. A neuron with a small receptive field sees only local texture and cannot understand global object structure. Deep networks are designed so that early layers have small receptive fields (local edges), and later layers have large receptive fields (entire objects), creating the hierarchical feature representation that makes CNNs powerful. + +### Related Concepts +Deep Learning, Transfer Learning + +## RNN & LSTM +**Definition:** Recurrent Neural Networks process sequential data by maintaining a hidden state that captures information from previous time steps. Long Short-Term Memory networks extend RNNs with gating mechanisms that enable learning long-term dependencies by controlling information flow through a cell state. + +### Theory & Explanation + +Recurrent Neural Networks (RNNs) are designed for sequential data — text, time series, audio, video — where each data point depends on previous ones. Unlike feedforward networks that assume independent inputs, RNNs maintain a hidden state h_t that gets updated at each time step: h_t = f(W·h_{t-1} + U·x_t + b), where x_t is the current input, h_{t-1} is the previous hidden state, and W, U, b are learned parameters. The same weight matrices are reused at every time step (parameter sharing), which allows the network to process sequences of any length and learn patterns regardless of position in the sequence. This is analogous to a convolutional filter being applied across spatial positions — here, the recurrent weights are applied across time steps. The output at each step can be computed from h_t, enabling applications like language modeling (predict next word) and sequence labeling (part-of-speech tagging). + +Despite their elegant design, vanilla RNNs have a fundamental limitation: vanishing gradients in time. During backpropagation through time (BPTT), gradients are multiplied by the same recurrent weight matrix at each time step. If the eigenvalues of this matrix are less than 1, gradients shrink exponentially with sequence length. This means vanilla RNNs can reliably learn dependencies spanning only about 5-10 time steps. For longer dependencies — like a pronoun referring to a subject 20 words earlier — the gradient signal essentially vanishes, and the RNN cannot connect them. Exploding gradients are also possible (eigenvalues > 1), but these are easier to detect and fix with gradient clipping. This fundamental limitation motivated the development of gated architectures. + +LSTM (Long Short-Term Memory) networks, introduced by Hochreiter and Schmidhuber in 1997, solve the vanishing gradient problem with a carefully designed gating mechanism. The core innovation is the cell state C_t, which acts like a conveyor belt running through the entire sequence with only minor linear interactions. Information can be added to or removed from the cell state through three gates. The forget gate (f_t = σ(W_f·[h_{t-1}, x_t] + b_f)) decides what information to discard from the previous cell state — it looks at the current input and previous hidden state and outputs a value between 0 (completely forget) and 1 (completely keep). The input gate (i_t = σ(W_i·[h_{t-1}, x_t] + b_i)) decides what new information to store, while a candidate cell state candidate \~C_t = tanh(W_C·[h_{t-1}, x_t] + b_C) creates the potential new values. The cell state is updated: C_t = f_t ⊙ C_{t-1} + i_t ⊙ \~C_t. Finally, the output gate (o_t = σ(W_o·[h_{t-1}, x_t] + b_o)) controls what parts of the cell state to output as the hidden state: h_t = o_t ⊙ tanh(C_t). + +The key to LSTM's ability to learn long-term dependencies lies in the cell state gradient flow. The gradient of C_t with respect to C_{t-1} is the forget gate activation f_t. If the forget gate is near 1 (remember everything), the gradient passes through almost unchanged, even across hundreds of time steps. This additive gradient path — the cell state update involves adding (not multiplying) new information — prevents the exponential gradient decay that plagues vanilla RNNs. The sigmoid gates (which output values between 0 and 1) control the flow, and because they are learned, the network can decide what to remember based on the data. For example, in language modeling, the forget gate learns to keep gender information until a pronoun needs it, then update when the subject changes. + +GRU (Gated Recurrent Unit) is a simplified variant with just two gates: the update gate (combining forget and input gates) and the reset gate. GRUs have fewer parameters than LSTMs (no separate cell state, fewer weight matrices) and often perform comparably on many tasks. The update gate decides how much of the past information to pass to the future, while the reset gate determines how much past information to forget. GRUs are computationally more efficient and require less data to generalize, making them suitable for smaller datasets. LSTMs tend to perform better on tasks requiring very long-term dependencies or when data is abundant. Both have been largely superseded by Transformers for NLP tasks, but they remain relevant for time series forecasting, small-scale sequence modeling, and scenarios where computational resources are constrained. + +Bidirectional RNNs/LSTMs process sequences in both forward and backward directions, with two separate hidden states concatenated at each time step. This provides full context from both past and future — critical for tasks like named entity recognition where the word after an entity can help classify it. The forward LSTM reads the sequence left to right, the backward LSTM reads right to left, and their outputs are combined. This doubles the parameter count but significantly improves performance on any task where complete context is beneficial. + +### Example + +Consider language modeling for the sentence: "I grew up in France. My parents moved there when I was young. Now, I speak fluent ___." To predict the final word, the model needs to remember "France" from 15+ words ago. With a vanilla RNN, the gradient signal from the final position would diminish to near zero by the time it reaches "France", making it impossible to learn this connection. With an LSTM, the forget gate learns to preserve country-related information in the cell state: when processing "France", the input gate writes this information to the cell state. During the intervening words ("My parents moved there when I was young"), the forget gate stays close to 1, preserving the cell state content. At "Now, I speak fluent", the output gate reads the relevant cell state information and uses it to predict "French". The gradient from the final prediction can flow back through the cell state connection with minimal attenuation, allowing the model to learn that "France" predicts "French" even across long distances. + +### Interview Questions + +**Q: What specific problem does LSTM solve that vanilla RNN cannot?** +A: LSTM solves the vanishing gradient problem in long sequences. Vanilla RNNs apply the same weight matrix multiplication at each time step during backpropagation through time. If this matrix's largest singular value is less than 1, the gradient signal shrinks exponentially with sequence length, making it impossible to learn dependencies beyond about 5-10 steps. LSTMs solve this through the cell state, which provides an additive gradient highway: the cell state update is C_t = f_t ⊙ C_{t-1} + i_t ⊙ \~C_t, where f_t is the forget gate. The gradient of C_t with respect to C_{t-1} is f_t (element-wise multiplication), not a matrix multiplication. When the forget gate is near 1 (which is learned), gradients flow backward through hundreds of time steps essentially unchanged. Additionally, the additive update means new information is added to the cell state rather than overwriting it, further stabilizing gradients. This architectural difference allows LSTMs to learn dependencies over 100+ time steps, while vanilla RNNs are limited to approximately 5-10 steps in practice. + +**Q: What are the three gates in LSTM and what does each do?** +A: The three gates are the forget gate, input gate, and output gate. The forget gate (f_t = σ(W_f·[h_{t-1}, x_t] + b_f)) decides what information to discard from the previous cell state. It examines the current input and previous hidden state, outputting values between 0 and 1 for each element of the cell state — 0 means completely forget that piece of information, 1 means completely retain it. In practice, the forget gate learns what context to keep (e.g., preserve the subject's gender until a pronoun is needed) and what to discard (e.g., reset when a new sentence starts). The input gate (i_t = σ(W_i·[h_{t-1}, x_t] + b_i)) controls what new information to store in the cell state. It works together with a candidate cell state created by a tanh layer: the input gate decides which values to update, and the tanh creates the candidate values. Together they add relevant new information to the ongoing cell state. The output gate (o_t = σ(W_o·[h_{t-1}, x_t] + b_o)) controls what information from the cell state is exposed as the hidden state at the current time step. Since the cell state may contain information not immediately needed, the output gate selects a relevant subset to pass to the next layer and to the next time step's hidden state. + +**Q: GRU vs LSTM — when would you use each?** +A: Use LSTM when: (1) you have abundant training data and need to capture very long-range dependencies (50+ time steps), (2) you are working on tasks where fine-grained control over memory (separate forget and input gates) matters, such as music generation or complex language modeling, (3) you have the computational budget for training since LSTMs have 3 gates vs GRU's 2, meaning about 25% more parameters. Use GRU when: (1) your dataset is smaller — GRU's fewer parameters reduces overfitting risk and requires less data to converge, (2) computational efficiency or inference speed is critical, (3) you are prototyping and want faster training iterations, (4) the task does not require extremely long-range dependencies — GRUs often match LSTM performance on many standard benchmarks with less computation. In practice, the performance gap is often small, and many practitioners try GRU first for computational efficiency. However, for production systems where every percentage point of accuracy matters and data is abundant, LSTM's additional capacity often edges out GRU. + +### Related Concepts +RNN, GRU, Sequence Modeling, Transformer + +## Transformers +**Definition:** A neural network architecture based entirely on self-attention mechanisms, eliminating recurrence and processing all tokens in parallel. The Transformer is the foundation of modern NLP and the backbone of all major language models including BERT, GPT, T5, and their successors. + +### Theory & Explanation + +The Transformer architecture, introduced in the landmark 2017 paper "Attention Is All You Need" by Vaswani et al., fundamentally changed how neural networks process sequences. Instead of processing tokens one by one like RNNs, Transformers use self-attention to compute representations that capture relationships between all pairs of tokens simultaneously. This parallelization enables training on orders of magnitude more data than was previously feasible — GPT-3 was trained on 570 GB of text with 175 billion parameters, a scale that would have been impossible with recurrent architectures. The Transformer has become the universal building block across NLP, computer vision (Vision Transformers), speech processing, and multi-modal models. + +The core mechanism of the Transformer is scaled dot-product attention. Given an input sequence, each token is projected into three vectors: Query (Q), Key (K), and Value (V) through learned linear transformations. The attention score between two tokens is computed as the dot product of their Query and Key vectors, which measures how much the first token should attend to the second. These scores are scaled by 1/√d_k (where d_k is the key dimension) to prevent the dot products from growing too large and pushing softmax into regions of extremely small gradients. A softmax function converts the scores into attention weights (a probability distribution), and the final output is a weighted sum of the Value vectors. Mathematically: Attention(Q,K,V) = softmax(Q·K^T/√d_k)·V. For the sentence "The cat sat on the mat", computing attention for "sat" involves comparing its Query with the Key of every other token, producing weights that indicate "cat" (subject) is most relevant, followed by "on" and "mat". + +Multi-head attention runs multiple attention operations in parallel (typically 8 to 16 heads), each with different learned projections for Q, K, and V. Different heads learn to capture different types of relationships — one head might capture syntactic dependencies (subject-verb), another captures positional proximity, another captures semantic relationships, and yet another captures coreference (pronouns linking to nouns). The outputs from all heads are concatenated and linearly projected back to the model dimension. Multi-head attention allows the model to attend to different types of information simultaneously from multiple representation subspaces, significantly increasing the model's representational capacity without increasing computational complexity per head. + +The Transformer architecture consists of an encoder and a decoder, each built from stacked identical layers (typically 6 to 96 layers). Each encoder layer has two sub-layers: multi-head self-attention and a position-wise feedforward network (two linear transformations with a ReLU activation in between). Each sub-layer has a residual connection (adding the input back to the output) followed by layer normalization: output = LayerNorm(x + Sublayer(x)). The decoder has three sub-layers: masked self-attention (prevents attending to future tokens), cross-attention (attends to encoder output), and the feedforward network. The masked self-attention in the decoder ensures autoregressive generation — when predicting the next word, it can only attend to previous words, not future ones. The cross-attention connects the decoder to the encoder, using encoder output as Keys and Values while the decoder provides Queries. + +Since self-attention has no inherent notion of token position (unlike RNNs which are inherently sequential), Transformers use positional encoding to inject sequence order information. The original paper used sinusoidal position encodings with different frequencies, allowing the model to learn relative positions. Modern implementations typically learn positional embeddings during training or use Rotary Position Embeddings (RoPE) and ALiBi (Attention with Linear Biases), which have been shown to improve length generalization — the ability to handle sequences longer than those seen during training. The positional information allows the model to distinguish between "The cat sat on the mat" and "The mat sat on the cat" — without it, the two sentences would produce identical representations since bag-of-words order is lost in self-attention. + +Transformers have evolved into two dominant paradigms. Encoder-only models (BERT, RoBERTa) use bidirectional self-attention and excel at understanding tasks: classification, question answering, named entity recognition, and any task requiring full context. Decoder-only models (GPT-3, GPT-4, Llama, Claude) use autoregressive (causal) attention and excel at generation tasks: text completion, dialogue, code generation, and creative writing. Encoder-decoder models (T5, BART) combine both for sequence-to-sequence tasks: translation, summarization, and text rewriting. The architectural choice depends on the downstream task, but decoder-only models have become dominant in recent years due to their scaling properties and ability to perform in-context learning after sufficient pretraining. + +### Example + +Consider the sentence "The cat sat on the mat" and compute self-attention for the word "sat". The Query for "sat" is compared with the Key of every token via dot product: Q_sat·K_The, Q_sat·K_cat, Q_sat·K_sat, Q_sat·K_on, Q_sat·K_the, Q_sat·K_mat. These raw scores might be [5, 15, 10, 8, 4, 7] (the values depend on learned projections). After scaling by 1/√d_k (say d_k=64, so 1/√64=1/8), the scores become [0.625, 1.875, 1.25, 1.0, 0.5, 0.875]. Softmax converts these to probabilities: [0.08, 0.40, 0.18, 0.14, 0.06, 0.14] — "cat" gets the highest attention weight (0.40) because it is the subject of "sat", followed by "sat" itself (0.18) and positional neighbors. The final representation of "sat" is the weighted sum of all Value vectors: V_sat(output) = 0.08·V_The + 0.40·V_cat + 0.18·V_sat + 0.14·V_on + 0.06·V_the + 0.14·V_mat. Critically, all these computations happen simultaneously for every token in the sequence — the attention scores for all token pairs are computed as a single matrix multiplication Q·K^T, making the process highly parallelizable on GPUs. + +### Interview Questions + +**Q: Why is self-attention better than recurrence for sequence modeling?** +A: Three major advantages. First and most important, parallelization: RNNs process tokens sequentially — token t cannot be processed until token t-1 is done — which prevents leveraging GPU parallelism. Transformers compute all token representations simultaneously using matrix multiplications, enabling orders of magnitude faster training on modern hardware. Second, long-range dependencies: RNNs suffer from vanishing gradients and struggle with dependencies beyond 10-20 steps. Self-attention provides direct connections between any two tokens regardless of distance — the path length is O(1) for attention versus O(n) for RNNs — making it trivial to learn long-range relationships. Third, computational efficiency per layer: for sequence length n and embedding dimension d, self-attention is O(n²·d), while RNN is O(n·d²). For typical NLP tasks where n is smaller than d² (e.g., n=512, d=768, d²=589,824), attention is more efficient. The quadratic O(n²) cost of attention is a limitation for very long sequences (10K+ tokens), addressed by sparse attention patterns. + +**Q: Explain the Query, Key, Value mechanism in self-attention.** +A: The Query-Key-Value mechanism is inspired by information retrieval systems. Think of Keys as labels attached to each token describing what information it contains, and Queries as what each token is looking for. The attention score is computed by matching Queries against Keys via dot product — a high score means a token's Query finds relevant information in another token's Key. Values are the actual content that will be aggregated. For a concrete analogy: imagine you (Query) are searching a library catalog (Keys) for books about "cats". The catalog entry "The Cat in the Hat" has a high match because "cat" appears in its keywords, so you retrieve that book (its Value is weighted highly). "The History of Rome" has a low match, so its Value is weighted near zero. In the Transformer, each token projects its input embedding into three separate spaces via learned weight matrices W_Q, W_K, W_V. The Query asks "what information do I need?", the Key answers "what information do I contain?", and the Value provides "here is the actual information if relevant". Different tokens attend to different information — "sat" queries for its subject and finds "cat", while "mat" queries for its location context. + +**Q: What are the advantages of multi-head attention over single-head attention?** +A: Multiple heads allow the model to attend to different types of information simultaneously from different representation subspaces. With single-head attention, the mechanism must capture all types of relationships — syntactic, semantic, positional, coreferential — within one set of Q/K/V projections, which forces compromises. Multi-head attention removes this bottleneck: each head learns distinct projection matrices, allowing specialization. Empirically, different heads learn complementary patterns: one head focuses on syntactic dependencies (verbs attending to their subjects), another captures positional proximity (nearby words attending to each other), another handles coreference (pronouns attending to their antecedents), and yet another may capture rare or domain-specific patterns. Mathematically, with h heads of dimension d_k = d_model/h, the total computation is equivalent to a single head of dimension d_model (since concatenating h heads of dimension d_k gives dimension d_model), but the representations are more expressive because each head operates on its own learned projection. Multi-head attention also provides robustness: if one head fails to learn useful patterns, others can compensate. + +### Related Concepts +Attention, BERT, GPT, LLMs + +--- + +# 4. NATURAL LANGUAGE PROCESSING + +## Tokenization +**Definition:** The process of breaking text into smaller units called tokens — words, subwords, or characters — that serve as the fundamental input units for NLP models. Tokenization is the first and essential preprocessing step for any NLP system. + +### Theory & Explanation + +Tokenization converts raw text into a sequence of discrete tokens that the model can process. The choice of tokenization level (word, character, subword) involves a fundamental tradeoff between vocabulary size, sequence length, and the ability to handle out-of-vocabulary (OOV) words. Word-level tokenization splits text on whitespace and punctuation, producing tokens like "The", "transformer", "revolutionized", "NLP". It is simple and intuitive, with each token carrying clear semantic meaning. However, it creates very large vocabularies (500K+ in English), suffers from OOV issues for rare words, misspellings, and novel terms, and treats morphologically related words ("run", "runs", "running", "ran") as entirely separate tokens, failing to capture their shared meaning. For morphologically rich languages like Turkish or German, word-level tokenization leads to vocabularies in the millions. + +Character-level tokenization processes each character as a token ("T", "h", "e", " ", "t", "r", ...). It completely eliminates OOV issues — any string can be represented as a sequence of known characters — and has a tiny vocabulary (26 letters + digits + punctuation = ~100 tokens). However, character-level models must process very long sequences (a 100-word sentence becomes 500+ character tokens), making them computationally expensive and requiring the model to learn word boundaries and spelling patterns from scratch. Characters carry minimal semantic information individually; the model must aggregate many characters to understand even simple words. Character-level models have been successful in some settings (character-level RNNs for text generation) but are rarely used in modern Transformer architectures due to sequence length constraints. + +Subword tokenization achieves a practical balance between word and character levels. The core idea is to represent common words as single tokens and break rare words into smaller, meaningful subword units. The most widely used subword algorithms are Byte Pair Encoding (BPE), WordPiece, and SentencePiece. BPE starts with a base vocabulary of individual characters and iteratively merges the most frequent pair of adjacent tokens in the training corpus. For example, if "th" and "e" are the most frequent adjacent pair, they are merged into "the", and the process repeats. After 10,000-100,000 merges, the final vocabulary contains common words ("the", "transformer", "revolution") and common subword units ("tion", "ing", "pre"). New words are encoded using these learned units: "Transformers" becomes "Transform" + "ers", handling the morphological variation naturally. + +WordPiece, used in BERT, is similar to BPE but differs in how it selects merges. Instead of merging the most frequent pair, WordPiece merges pairs that maximize the likelihood of the training data, computed as the pair's frequency divided by the product of individual token frequencies. This tends to create more linguistically meaningful subword units. SentencePiece, used in T5, XLM-R, and Llama 2, takes a different approach: it treats the input as raw Unicode bytes rather than requiring pre-tokenized text (no space splitting needed). It applies either BPE or unigram language model tokenization directly on the raw character sequence, which makes it language-agnostic and capable of handling any language without modification. SentencePiece also handles whitespace as a regular character (typically marked with "▁" prefix), preserving the ability to reconstruct original text. + +The vocabulary size in subword tokenization is typically 30,000 to 100,000 tokens. GPT-2 uses 50,257 tokens, BERT uses 30,000, Llama 2 uses 32,000, and GPT-4 uses over 100,000. Smaller vocabularies mean fewer parameters (the embedding matrix accounts for a significant portion of model parameters) but longer tokenized sequences. The tradeoff is important: a 100,000-token vocabulary requires a 100,000×d_model embedding matrix, which can be hundreds of millions of parameters. However, it also means more text can be represented per token (English averages about 0.75 words per token with GPT-4's tokenizer), reducing sequence length and computational cost. Most modern models use some form of subword tokenization, and this standardization has been crucial for enabling large-scale pretraining across diverse languages and domains. + +### Example + +Consider the sentence "The transformer revolutionized NLP in 2017!" processed at different tokenization levels. Word-level tokenization splits on whitespace and punctuation: ["The", "transformer", "revolutionized", "NLP", "in", "2017", "!"] — 7 tokens. The vocabulary would need to contain "revolutionized" (lower frequency) and "2017" (specific year, won't generalize). BPE tokenization (GPT-4's tokenizer approximately): ["The", "▁trans", "former", "▁revolution", "ized", "▁N", "LP", "▁in", "▁2017", "!"] — 10 tokens. Note how "transformer" is split into "trans" + "former" (both frequent subwords), "revolutionized" into "revolution" + "ized" (captures morphological pattern -ized used across many words), and "NLP" into "N" + "LP". The word "The" is kept whole (common word), and numbers like "2017" are kept as single tokens. Character-level: ["T", "h", "e", " ", "t", "r", "a", "n", "s", "f", "o", "r", "m", "e", "r", " ", "r", "e", "v", "o", "l", "u", "t", "i", "o", "n", "i", "z", "e", "d", " ", "N", "L", "P", " ", "i", "n", " ", "2", "0", "1", "7", "!"] — 42 tokens. The model must learn from scratch that "t+r+a+n+s+f+o+r+m+e+r" forms a single concept. + +### Interview Questions + +**Q: Why is subword tokenization used in all modern large language models?** +A: Subword tokenization provides the optimal balance between vocabulary size, sequence length, and OOV handling. Word-level tokenization has three fatal flaws for LLMs: (1) massive vocabulary sizes (500K+) that make the embedding layer prohibitively large, (2) inability to handle rare words, misspellings, or novel terms (OOV), and (3) failure to capture morphological relationships between words like "run" and "running". Character-level tokenization avoids OOV entirely but creates very long sequences (500+ tokens per sentence), making self-attention O(n²) computationally infeasible for long text. Subword tokenization splits rare words into common subword units ("revolutionized" → "revolution" + "ized"), keeps common words as single tokens, handles any input without OOV errors, maintains reasonable sequence lengths (about 1.3 tokens per word in English), and achieves manageable vocabulary sizes (30K-100K). This standardization enabled the scaling of LLMs because models can process diverse text (technical terms, code, multiple languages, misspellings) with a fixed, manageable vocabulary and learn useful compositionality from subword boundaries. + +**Q: What are the tradeoffs between a smaller and larger vocabulary size in subword tokenization?** +A: Smaller vocabulary (30K-50K tokens): pros include a smaller embedding matrix (fewer parameters, especially important for large models), denser token representations (each token seen more frequently, better learned), and lower memory usage. Cons include longer tokenized sequences (more tokens per sentence), which increases self-attention O(n²) computation cost and may reduce effective context length. Larger vocabulary (100K+ tokens): pros include shorter sequences (fewer tokens per sentence, faster attention, more effective context), better encoding efficiency (tokens carry more meaning per unit), and better handling of domain-specific terms as single tokens. Cons include a much larger embedding matrix (significant parameter overhead), sparser token frequencies (many tokens rarely seen), and potentially worse generalization for rare tokens. The optimal size depends on the model's scale, available compute, and target applications. GPT-4 uses a very large vocabulary (~100K) prioritizing sequence compression, while smaller models often use 32K vocabularies to keep total parameters manageable. + +### Related Concepts +BPE, Word Embeddings, BERT, GPT + +## Bag of Words & TF-IDF +**Definition:** Bag of Words (BoW) represents a document as a multiset of its words, ignoring grammar and word order, creating a fixed-length vector of word frequencies. TF-IDF (Term Frequency-Inverse Document Frequency) extends BoW by weighting words based on their importance across the corpus, downweighting common words and upweighting rare, informative ones. + +### Theory & Explanation + +Bag of Words is one of the simplest and most fundamental text representation techniques. Given a vocabulary of size V (all unique words in the corpus), each document is represented as a V-dimensional vector where each dimension counts the occurrences of a specific word. For example, with vocabulary ["cat", "dog", "mat", "sat"], the document "the cat sat on the mat" becomes [1, 0, 1, 1] (assuming stop words "the" and "on" are removed). This representation is extremely sparse — most documents use only a small fraction of the total vocabulary. The advantages are simplicity (just counting), interpretability (each dimension corresponds to a specific word), and efficiency (can use sparse matrix operations). The limitations are equally significant: complete loss of word order and context ("cat bit dog" and "dog bit cat" are identical), inability to capture semantics ("excellent" and "good" are independent dimensions even though they are related), and high dimensionality with data sparsity. + +N-grams extend BoW to capture short word sequences, preserving some local context. A unigram model uses single words, bigrams use word pairs ("the cat", "cat sat", "sat on"), and trigrams use triplets. This captures phrase-level information — "not good" and "very good" become distinct features, and "New York" is treated as a single phrase rather than two separate words. However, n-grams explode the feature space (vocabulary grows exponentially with n) and suffer from severe sparsity (most n-grams occur only once). In practice, n-grams beyond 3 are rarely used with BoW, and feature selection or hashing tricks are needed to manage dimensionality. Despite its simplicity, BoW with n-grams remains a strong baseline for text classification, often competitive with simple neural approaches. + +TF-IDF addresses BoW's problem of common words dominating the representation. Words like "the", "is", "at" appear in almost every document but carry little discriminative information. TF-IDF assigns each word w in document d a weight that combines: Term Frequency (TF) = (number of times w appears in d) / (total words in d), and Inverse Document Frequency (IDF) = log(total documents / number of documents containing w). The TF-IDF score = TF × IDF. Common words have high TF but low IDF (they appear in most documents, so log(N/df) is near 0), giving them low weight. Words that appear frequently in a specific document but rarely in the corpus (like a document's key topic) get high TF-IDF scores, accurately reflecting their importance for that document. The IDF formula with log smoothing prevents division by zero and provides diminishing returns for extremely rare words. + +Stop word removal (filtering common words like "the", "a", "is") is a common preprocessing step for both BoW and TF-IDF. However, many modern implementations skip explicit stop word removal because TF-IDF naturally handles them through low IDF values. Additional preprocessing includes stemming (reducing words to root form: "running", "runs", "ran" → "run") and lemmatization (dictionary-based root: "better" → "good"). Stemming is faster but can produce non-words, while lemmatization is more accurate but requires a dictionary. Both reduce vocabulary size and improve generalization by collapsing morphological variants. + +Despite being largely superseded by word embeddings and transformers for high-stakes NLP tasks, BoW and TF-IDF remain relevant in several contexts. They are excellent baselines — if a simple TF-IDF + logistic regression model cannot outperform a complex neural approach on a task, there may be fundamental data issues. They provide interpretable features that can be inspected and explained to stakeholders. They work well with limited data where neural models cannot learn meaningful representations. And they are used in information retrieval (search engines use TF-IDF variants like BM25 as core ranking signals). At EY L1 level, understanding these foundational techniques demonstrates awareness of the NLP landscape and the ability to choose appropriate methods for different constraints. + +### Example + +Consider three documents in a corpus: D1 = "cat sat on mat", D2 = "dog sat on log", D3 = "I love cats and dogs as pets". With vocabulary ["cat", "sat", "mat", "dog", "log", "love", "cats", "dogs", "pets"] after removing stop words. BoW representations: D1 = [1,1,1,0,0,0,0,0,0], D2 = [0,1,0,1,1,0,0,0,0], D3 = [0,0,0,0,0,1,1,1,1]. TF-IDF for "mat" in D1: TF = 1/3 (one occurrence in 3 words), IDF = log(3/1) = log(3) = 0.477, TF-IDF = 0.333 × 0.477 = 0.159. For "sat" in D1: TF = 1/3, IDF = log(3/2) = 0.176, TF-IDF = 0.333 × 0.176 = 0.059. Note that "mat" has a higher TF-IDF in D1 than "sat" because "mat" is unique to D1 (appears in only 1 document) while "sat" appears in 2 documents. For "dog" in D2: IDF = log(3/1) = 0.477, TF = 1/3, TF-IDF = 0.159 — same high score because it is also unique. The TF-IDF approach correctly identifies "mat" and "dog" as more discriminative terms than the shared word "sat". + +### Interview Questions + +**Q: What are the key limitations of the Bag of Words representation?** +A: Six major limitations. First, loss of word order and syntax: "cat bit dog" and "dog bit cat" produce identical BoW vectors despite opposite meanings — BoW cannot distinguish subject from object. Second, semantic blindness: synonyms like "car" and "automobile" are treated as completely independent dimensions, missing their semantic similarity, while homographs like "bank" (river vs financial) are counted as the same word regardless of meaning. Third, high dimensionality and sparsity: with a vocabulary of 50,000 words, each document vector has 50,000 dimensions but most entries are zero, making the representation storage-inefficient and causing the curse of dimensionality for learning algorithms. Fourth, inability to handle OOV words: any word not in the training vocabulary is simply ignored during inference. Fifth, no contextual meaning: the same word always contributes the same weight regardless of context — "bank" in "river bank" and "investment bank" are treated identically. Sixth, the representation does not capture phrase-level meaning without explicitly adding n-gram features, which further explode dimensionality. These limitations motivated the development of dense embeddings and contextual representations. + +**Q: How does TF-IDF improve over simple Bag of Words?** +A: TF-IDF addresses BoW's core problem of common but uninformative words dominating the representation. In pure BoW, the most frequent words in a document are typically stop words ("the", "is", "and"), which contribute the largest values to the vector even though they carry no topical information. TF-IDF corrects this by multiplying term frequency by inverse document frequency, which heavily downweights words that appear in many documents. For example, "the" might have TF = 3 in a document but IDF = log(N/N) = 0 (or very small) since it appears in every document, giving it near-zero final weight. Conversely, a specialized term like "convolutional" in a machine learning paper gets high IDF since it appears in only a few documents, so even if it appears once, its TF-IDF weight is high. This weighting scheme ensures that the most discriminative, topic-specific words dominate the document representation. TF-IDF also naturally handles corpus-level word importance — a word common in one domain but rare globally (e.g., "qubit" in quantum computing papers) gets appropriately high weight within that domain while being irrelevant elsewhere. + +**Q: TF-IDF vs word embeddings — when would you use each?** +A: Use TF-IDF when: (1) you have limited training data (hundreds to low thousands of examples), where neural embeddings cannot learn meaningful representations; (2) interpretability is critical — each TF-IDF feature directly corresponds to a specific word, making it easy to explain model decisions to stakeholders or regulators; (3) you need a quick, compute-efficient baseline that runs on a laptop without GPUs; (4) you are working on document retrieval or search ranking where TF-IDF variants like BM25 remain state-of-the-art components; (5) you need sparse, high-dimensional representations suitable for linear models or traditional ML pipelines. Use word embeddings when: (1) you have sufficient data (tens of thousands of examples or more) to learn dense representations; (2) semantic similarity matters — embeddings capture that "car" and "automobile" are close, enabling generalization; (3) you are building deep learning models that can learn end-to-end from raw text; (4) you need pre-trained representations that transfer across tasks; (5) your application involves rare words or OOV handling where subword embeddings like FastText help. A practical approach often combines both: use TF-IDF for feature engineering in traditional ML pipelines and embeddings for neural approaches, or concatenate both as features. + +### Related Concepts +Word Embeddings, Text Preprocessing + +## Word Embeddings +**Definition:** Dense, low-dimensional vector representations of words where semantically similar words map to nearby points in the embedding space. Unlike sparse BoW/TF-IDF representations, word embeddings capture rich semantic and syntactic relationships through their geometric arrangements. + +### Theory & Explanation + +The fundamental idea underlying word embeddings is the distributional hypothesis, popularized by linguist John Firth in 1957: "You shall know a word by the company it keeps." This means words that appear in similar contexts have similar meanings. "Dog" and "puppy" appear in similar contexts ("the ___ ran", "I walked my ___, "___ is friendly"), so they should have similar representations. "Dog" and "car" share fewer contexts, so they should be further apart. Word embeddings operationalize this hypothesis by learning a dense vector (typically 50-300 dimensions) for each word such that the vectors of co-occurring words are close together. The result is a continuous, low-dimensional vector space where geometric relationships encode semantic and syntactic regularities. + +Word2Vec, introduced by Mikolov et al. in 2013, is the most influential word embedding method. It comes in two architectures. CBOW (Continuous Bag of Words) predicts a target word from its surrounding context words. For the sentence "The cat sat on the mat", with a window of 2, CBOW uses ["The", "cat", "on", "the"] to predict "sat". The model is a simple neural network with one hidden layer (the embedding lookup), trained to maximize the probability of the target word given the context. Skip-gram does the opposite: given a target word, it predicts the surrounding context words. Given "sat", it predicts ["The", "cat", "on", "the"]. Skip-gram is slower to train but produces better representations for rare words, while CBOW is faster and works well for frequent words. Both architectures are efficient enough to train on billions of words, producing high-quality embeddings as a byproduct of the prediction task. + +GloVe (Global Vectors for Word Representation) takes a different approach. While Word2Vec predicts words based on local contexts, GloVe leverages global word co-occurrence statistics across the entire corpus. It constructs a word-word co-occurrence matrix X where X_ij counts how often word j appears in the context of word i. GloVe then learns embeddings such that the dot product of word i and word j vectors approximates the logarithm of their co-occurrence probability: w_i · w_j + b_i + b_j ≈ log(X_ij). This combines the benefits of global matrix factorization (like LSA) with the local context learning of Word2Vec. GloVe embeddings often capture semantic relationships more accurately than Word2Vec because they are trained on global corpus statistics rather than stochastic window-based predictions. + +The most celebrated property of word embeddings is their ability to capture analogies through vector arithmetic. The classic example: vector("king") — vector("man") + vector("woman") ≈ vector("queen"). This works because the embedding space organizes along semantic axes: the vector from "man" to "woman" represents the gender dimension, and applying this same transformation to "king" moves it to "queen". Similar analogies work for verb tenses ("walk":"walked" :: "run":"ran"), capital-city relationships ("Paris":"France" :: "Berlin":"Germany"), and comparative adjectives ("big":"bigger" :: "small":"smaller"). These analogies emerge purely from distributional statistics — the model has never been explicitly taught about gender, tense, or geography. This property makes embeddings powerful for transfer learning: a model pretrained on massive text can be applied to downstream tasks with minimal task-specific data. + +Static embeddings (Word2Vec, GloVe, FastText) assign a single fixed vector to each word regardless of context. The word "bank" has the same embedding whether it appears in "river bank" or "investment bank". This is a significant limitation — polysemy and context-dependent meaning are not captured. Contextual embeddings, introduced by models like ELMo, BERT, and GPT, solve this by computing word representations dynamically based on surrounding context. "Bank" in "river bank" gets a different representation than in "investment bank", and these representations encode fine-grained, context-specific meaning. For this reason, contextual embeddings have largely replaced static embeddings in modern NLP systems. However, static embeddings remain important for efficiency (they are much faster to compute), for applications with limited computational resources, and as pre-trained features in traditional ML pipelines. FastText addresses one limitation of static embeddings by incorporating subword information: "eating" is represented as the sum of its character n-grams ["ea", "at", "ti", "in", "ng", "eat", "ati", ...], allowing it to handle morphologically complex words and OOV terms that previous static embeddings could not. + +### Example + +Consider a pre-trained GloVe embedding space with 100-dimensional vectors. Computing cosine similarity between word pairs: sim(dog, puppy) = 0.92 (both refer to canines, similar contexts), sim(dog, cat) = 0.75 (both common pets, share many contexts like "walk", "feed", "pet"), sim(dog, car) = 0.12 (very different contexts), sim(dog, computer) = -0.05 (essentially unrelated). For analogies: vector("paris") - vector("france") + vector("germany") produces the vector closest to vector("berlin"), with cosine similarity 0.78 to "berlin" versus 0.45 to the next nearest word "munich". Similarly, vector("king") - vector("queen") + vector("woman") produces a vector with nearest neighbors "man" (0.83), followed by "boy" (0.62). The difference vector "king" - "queen" approximately equals "man" - "woman", confirming that the gender dimension is consistent across different word pairs. This geometric structure is remarkable because it was not explicitly programmed — it emerged entirely from the distributional statistics of word co-occurrence patterns in the training corpus. + +### Interview Questions + +**Q: What is the fundamental difference between static and contextual word embeddings?** +A: Static embeddings (Word2Vec, GloVe, FastText) assign one fixed vector per word, regardless of context. The word "bank" has exactly the same embedding in "river bank", "investment bank", and "blood bank" — the model cannot distinguish these different senses. This limits performance on tasks requiring nuanced understanding of word meaning in context. Contextual embeddings (BERT, ELMo, GPT) compute a unique representation for each occurrence of a word based on its surrounding context. "Bank" in "I deposited money at the bank" gets a financial-institution representation (close to "banking", "financial"), while "bank" in "The river bank eroded" gets a geographical representation (close to "shore", "river"). Contextual embeddings achieve this using deep bidirectional or autoregressive models that process the entire input sentence. This contextual sensitivity leads to significantly better performance on virtually all NLP benchmarks — NER, QA, sentiment analysis, etc. The tradeoff is computational cost: static embeddings are a simple lookup (microseconds) while contextual embeddings require a full forward pass through a large neural network (milliseconds to seconds). For high-throughput or low-latency applications, static embeddings may still be preferred despite their lower accuracy. + +**Q: Explain the difference between Word2Vec CBOW and Skip-gram architectures.** +A: CBOW (Continuous Bag of Words) predicts a target word from its surrounding context words. Given a context window of words around a target, it averages the context word embeddings and tries to predict the target word. For "The cat sat on the mat" with window size 2, predicting "sat" uses ["The", "cat", "on", "the"] as input. CBOW is fast to train because it processes multiple context words at once and uses average pooling, making it about 2-3x faster than Skip-gram. It produces smoother embeddings that work well for frequent words. Skip-gram does the reverse: given a target word, it predicts each context word independently. Given "sat", the model tries to predict "The", "cat", "on", and "the" separately. This is a harder task (multiple predictions per input), making training slower but producing higher-quality embeddings, particularly for rare words. Skip-gram effectively treats each (target, context) pair as a separate training example, giving more weight to less frequent words since they still get multiple prediction opportunities. In practice, Skip-gram is generally preferred when training data is sufficient and embedding quality is critical, while CBOW is chosen for faster training or very large corpora. + +**Q: What is the distributional hypothesis and why is it important for word embeddings?** +A: The distributional hypothesis states that words that occur in similar contexts have similar meanings. Formally stated by John Firth in 1957: "You shall know a word by the company it keeps." This means the meaning of a word is defined by the contexts in which it appears, not by any inherent property. For example, "dog" and "puppy" appear in very similar contexts — "the ___ barked", "I walked my ___", "___ is a good boy" — and this contextual similarity is what makes them semantically related. The distributional hypothesis is the foundational principle behind all word embedding methods (Word2Vec, GloVe, FastText, BERT). These models all learn word representations by predicting words from their contexts or contexts from words, effectively encoding the distributional patterns in the training corpus into vector space geometry. The hypothesis has strong empirical support — embeddings trained on large corpora consistently capture synonymy, analogy, and semantic relationships, demonstrating that distributional similarity is a reliable proxy for semantic similarity. This approach has been so successful that it now dominates NLP, replacing manually curated lexical resources like WordNet. + +### Related Concepts +Word2Vec, GloVe, BERT, Sentence Embeddings + +## Sequence Models (RNN/LSTM/GRU/Attention/Transformer) +**Definition:** A family of neural network architectures designed to process sequential data where order matters — text, time series, speech, video. The evolution from RNN to Transformer represents the most significant technical progression in NLP, each step addressing limitations of its predecessor. + +### Theory & Explanation + +Sequence models are necessary because standard feedforward networks cannot handle variable-length sequences, capture temporal dependencies, or share parameters across sequence positions. The key insight across all sequence models is that information from previous time steps influences the processing of current and future time steps. This makes them fundamentally different from models that assume independent and identically distributed (i.i.d.) data. The evolution of sequence models can be understood as a progression of innovation: each new architecture solves a specific problem of the previous while introducing new capabilities, eventually leading to the Transformer dominance we see today. + +Recurrent Neural Networks (RNNs) were the first effective neural sequence model. They maintain a hidden state h_t that is updated at each time step: h_t = tanh(W·h_{t-1} + U·x_t + b). This hidden state acts as a memory, carrying information from earlier time steps to current processing. The same weight matrices W and U are reused at every step (parameter sharing), allowing the model to handle arbitrary sequence lengths. RNNs can model sequences of any length, accept variable-length inputs and outputs, and capture temporal dependencies. However, they suffer from vanishing gradients during backpropagation through time — the gradient signal exponentially decays with sequence length, limiting practical learning to about 5-10 steps. This is the fundamental limitation that LSTM and GRU were designed to overcome. + +LSTM and GRU address the vanishing gradient problem through gating mechanisms that control information flow. LSTMs introduce a separate cell state that acts as a gradient highway — information can flow through it with only linear transformations, preserving gradient magnitude across hundreds of steps. The forget, input, and output gates learn to control what to keep, what to add, and what to output. GRUs simplify this to two gates (reset and update) with fewer parameters. Both architectures can learn long-range dependencies of 100+ time steps, making them suitable for tasks like machine translation and document classification. Despite their success, they remain sequential in nature — each step must wait for the previous step's hidden state — which limits parallelization and thus training speed. + +Bidirectional RNNs extend the concept by processing sequences in both directions simultaneously. Two separate RNN layers (forward and backward) read the sequence left-to-right and right-to-left, and their hidden states are concatenated at each time step. This provides each token's representation with full context from both past and future. For example, in named entity recognition, knowing that "Apple" is followed by "Inc." helps classify it as an organization, while "eat an apple" classifies it as a fruit. Bidirectional processing significantly improves performance on any task where complete context is beneficial, at the cost of doubling parameters and requiring the entire sequence to be available (not suitable for real-time generation). + +The Attention mechanism was introduced to help sequence models focus on relevant parts of the input when producing each output. In encoder-decoder models (like machine translation), attention computes a context vector for each output position by taking a weighted sum of all encoder hidden states. The weights are learned — the decoder learns to "attend" to different source words when producing different target words. For English-to-French translation, when generating the French word for "ate", the model learns to attend most strongly to "ate" in the source sentence, while ignoring unrelated words. Attention dramatically improved performance on long sequences and provided interpretability through attention weight visualization. + +The Transformer took the attention concept to its logical conclusion: eliminate recurrence entirely and rely solely on attention mechanisms. By processing all tokens in parallel through self-attention, the Transformer achieves full parallelization (enabling training on unprecedented data scales), directly connects any two tokens regardless of distance (solving the long-range dependency problem), and scales to hundreds of billions of parameters. The quadratic O(n²) memory complexity of self-attention is the main limitation — a sequence of 100K tokens requires 10 billion attention computations — driving research into sparse attention patterns, linear attention, and hierarchical processing. As of 2025, Transformers are the dominant architecture for virtually all NLP tasks, with RNNs used primarily in niche applications like real-time streaming, embedded systems with limited compute, and certain time series forecasting problems. + +### Example + +Consider processing "The cat sat on the mat" with different architectures. RNN: processes token by token — step 1 processes "The", updates hidden state h1; step 2 processes "cat" with h1 to produce h2; step 3 processes "sat" with h2 to produce h3, and so on. Each step takes 100-200ms sequentially on CPU (hypothetical). Total time for 6 tokens: 6 sequential steps. The representation of "sat" (h3) has seen only ["The", "cat"] — it has no information about "on", "the", or "mat" yet. LSTM: same sequential processing but with a cell state that can maintain information about "cat" even as the sequence continues through "sat", "on", etc., enabling the model to connect "mat" back to "cat" through the cell state. Bidirectional LSTM: forward LSTM processes left-to-right, backward LSTM processes right-to-left. The final representation of "sat" concatenates forward h3 (seeing "The cat") and backward h3_rev (seeing "the mat on"), giving full context. Transformer: all 6 tokens are processed simultaneously. In one layer, every token directly attends to every other token — "sat" attends to "cat" (subject), "on" (preposition), "mat" (location). All computations happen as a single matrix multiplication on GPU, taking about the same time as one RNN step but processing the entire sequence. + +### Interview Questions + +**Q: Walk through the evolution from RNN to Transformers — why did each step improve?** +A: RNNs introduced sequential processing with parameter sharing across time, enabling variable-length sequence handling for the first time. However, vanishing gradients limited them to short sequences (~5-10 steps). LSTMs solved this with gating mechanisms (cell state + forget/input/output gates) that provide a gradient highway, extending reliable learning to 100+ steps. GRUs simplified LSTMs to two gates, reducing parameters while maintaining comparable performance. Bidirectional RNNs added future context by processing sequences both ways, which was crucial for understanding tasks like NER and classification where full context helps at every position. Attention addressed the bottleneck in encoder-decoder models where the final encoder hidden state had to compress the entire source sequence — attention allows the decoder to directly access all encoder states. Finally, the Transformer removed recurrence entirely, replacing it with self-attention. This enabled full parallelization (training on orders of magnitude more data), direct connections between any token pair (eliminating the distance bottleneck), and scalable to billions of parameters. Each step solved a specific limitation of the previous while maintaining the strengths, creating a clear evolutionary path. + +**Q: When would you still use an RNN or LSTM today instead of a Transformer?** +A: Despite Transformer dominance, RNNs/LSTMs remain relevant in specific scenarios. (1) Real-time streaming: RNNs process tokens one at a time with constant memory. For speech recognition or live transcription, you cannot wait for a complete sentence before producing output. RNNs naturally process input as it arrives. (2) Resource-constrained environments: on microcontrollers, embedded systems, or mobile devices, a small LSTM (1-5 MB) can run efficiently in real-time while a Transformer's memory and compute requirements are prohibitive. (3) Time series forecasting: for long time series (tens of thousands of steps), Transformer O(n²) memory becomes infeasible, while LSTM/GRU O(n) scaling handles them naturally. Specialized architectures like Informer or Autoformer address this but LSTMs remain competitive. (4) Very small data: LSTMs typically outperform Transformers when training data is limited (hundreds to low thousands of examples) because they have fewer parameters and stronger inductive biases. (5) Low latency requirements: applications requiring millisecond-level inference (algorithmic trading, interactive dialogue) benefit from LSTM's sequential but lightweight computation versus Transformer's full-sequence processing. + +**Q: What specific problem does the Attention mechanism solve?** +A: Attention solves the information bottleneck in encoder-decoder architectures. In the original RNN/LSTM encoder-decoder setup, the encoder compresses the entire source sequence (e.g., an entire sentence for translation) into a single fixed-size context vector (the final hidden state). This bottleneck forces the model to cram all source information — regardless of length — into one vector, which is impossible for long sentences. Empirically, translation quality degraded sharply for sentences longer than 20-30 words because the context vector could not retain all the information. Attention eliminates this bottleneck by allowing the decoder to access the entire sequence of encoder hidden states, not just the final one. When generating each output word, the decoder computes a weighted sum of all encoder states, with weights determined by relevance. For translating "The cat sat on the mat" to French "Le chat s'est assis sur le tapis", when generating "chat" the model attends most to "cat"; when generating "tapis" it attends to "mat". This means no information is lost through compression, the model can handle sentences of arbitrary length, and it gains interpretability through the attention weights that show which source words influenced which target words. Translation quality improved dramatically, especially for long sentences. + +### Related Concepts +LSTM, Transformer, Self-Attention + +## BERT & Pre-trained Language Models +**Definition:** BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model that learns deep bidirectional representations by conditioning on both left and right context simultaneously through masked language modeling. It set new benchmarks on 11 NLP tasks and established the pre-training then fine-tuning paradigm that dominates modern NLP. + +### Theory & Explanation + +BERT, introduced by Google in 2018, was a breakthrough that fundamentally changed NLP practice. Before BERT, NLP models were typically trained from scratch for each task, requiring large amounts of task-specific labeled data. BERT introduced a two-stage approach: pre-training on a massive unlabeled corpus (3.3 billion words from Wikipedia and BooksCorpus) using self-supervised objectives, followed by fine-tuning on a small labeled dataset for specific tasks. This transfer learning paradigm meant that a single pre-trained model could be adapted to dozens of tasks with minimal task-specific changes and limited labeled data. BERT's impact was immediate and dramatic — it improved the GLUE benchmark score from 72.8 to 80.4, and subsequent improvements extended this to 90+. + +The key architectural innovation of BERT is bidirectional pre-training. Previous language models like GPT were unidirectional (left-to-right), meaning each token could only attend to previous tokens. This is natural for generation but limits representational quality for understanding tasks — when understanding "bank" in "I deposited money at the bank", the model should consider both preceding words ("deposited money at the") and following words ("the bank") to determine it is a financial institution. BERT achieves bidirectionality through its Transformer encoder architecture and a novel pre-training objective: Masked Language Modeling (MLM). In MLM, 15% of input tokens are randomly masked (replaced with [MASK]), and the model must predict the original token based on the unmasked context from both directions. For "The cat [MASK] on the mat", BERT uses "The cat" (left context) and "on the mat" (right context) to predict "sat". This forces the model to build deep bidirectional representations. + +BERT's second pre-training objective is Next Sentence Prediction (NSP): given two sentences A and B, predict whether B follows A in the original text. This binary classification task teaches BERT to understand relationships between sentences, which is crucial for tasks like question answering (understanding if an answer follows a question) and natural language inference (determining if one sentence entails another). NSP uses the special [CLS] token output, which aggregates information from the entire sequence. In practice, subsequent work (RoBERTa) showed that NSP is not strictly necessary and removing it can improve performance on some tasks, but it was part of BERT's original design. + +Fine-tuning BERT for a downstream task requires minimal changes. The pre-trained model provides the encoder layers, and task-specific classification heads are added on top. For sequence classification (sentiment analysis), a classification layer is added to the [CLS] token output. For token classification (NER, POS tagging), a classification layer is applied to each token's output. For question answering, two vectors (start and end) predict the answer span boundaries. For sentence pair tasks (NLI), the two sentences are concatenated with a [SEP] token. All parameters (pre-trained + new) are fine-tuned end-to-end on the labeled data. Because BERT already understands language structure from pre-training, fine-tuning requires relatively little labeled data — 10,000 examples is often sufficient for strong performance, compared to 100,000+ for training from scratch. + +The BERT family has expanded significantly. RoBERTa (2019) showed that BERT was undertrained — by training longer, with larger batches, on more data (160 GB vs 16 GB), and with dynamic masking (different mask pattern each epoch), RoBERTa significantly outperformed the original BERT without architectural changes. DistilBERT uses knowledge distillation to create a model with 40% fewer parameters while retaining 95% of BERT's performance, making it suitable for production deployment. ALBERT reduces parameters through cross-layer parameter sharing and factorization of the embedding matrix. XLNet uses permutation language modeling (predicting tokens in random order) to combine the benefits of autoregressive and bidirectional models. T5 reframes all NLP tasks as text-to-text problems, using the same model architecture for translation, summarization, classification, and QA. BART combines BERT's bidirectional encoder with GPT's autoregressive decoder, making it particularly effective for text generation tasks like summarization. + +BERT's impact extends beyond NLP. The pre-training then fine-tuning paradigm has been adopted in computer vision (CLIP, DINO), speech recognition (Wav2Vec 2.0), and multi-modal learning (ViLT, LLaVA). The Transformer encoder architecture that BERT popularized is now used in recommendation systems, drug discovery, and code analysis. BERT also introduced several practical innovations that have become standard: the [CLS] token for classification, the [SEP] token for separating segments, learned segment embeddings, and the WordPiece tokenizer. For the EY L1 interview, understanding how BERT works, why it is bidirectional, and the pre-training/fine-tuning paradigm is the essential knowledge. + +### Example + +BERT fine-tuned for sentiment classification on movie reviews. The original BERT-base model has 12 Transformer encoder layers, 768 hidden dimensions, 12 attention heads, and 110M parameters. For sentiment analysis, we add a single linear classification layer on top of the [CLS] token: W (768×2 for positive/negative) + softmax. The input "This movie was absolutely fantastic! [SEP] I loved every minute." is tokenized by WordPiece into ["this", "movie", "was", "absolutely", "fantastic", "!", "[SEP]", "i", "loved", "every", "minute", "."]. The tokenized sequence passes through all 12 Transformer layers. The final [CLS] representation (vector of 768) captures the overall sentiment of the entire sequence. The classification layer maps this to {"positive": 0.92, "negative": 0.08}, predicting positive. Training on 10,000 movie reviews takes about 30 minutes on a single GPU and achieves 92% accuracy, compared to 85% for a TF-IDF + Logistic Regression baseline trained on the same data. The improvement comes from BERT's understanding of language nuance — it correctly handles negation ("not bad" does not mean "bad"), intensifiers ("absolutely fantastic" is more positive than "good"), and implicit sentiment ("I fell asleep" is negative without using negative words). + +### Interview Questions + +**Q: What is masked language modeling and why does BERT use it?** +A: Masked Language Modeling (MLM) randomly masks 15% of tokens in a sequence and trains the model to predict the masked tokens from their surrounding context. For example, given "The cat [MASK] on the [MASK]", the model must predict "sat" and "mat" based on the unmasked context from both directions. BERT uses MLM because it enables deep bidirectional representation learning. Previous models like GPT used left-to-right language modeling (predict the next word), which is inherently unidirectional — each token can only see previous tokens. This is fine for text generation but suboptimal for understanding tasks where context from both sides is crucial. MLM forces BERT to build representations that incorporate both left and right context simultaneously, producing richer token embeddings. The specific masking strategy: 80% of the time replace with [MASK], 10% replace with a random word, 10% keep unchanged. This mismatch between pre-training (where [MASK] appears) and fine-tuning (where [MASK] never appears) is handled by the random word replacement, which teaches the model to rely on context rather than assuming [MASK] is always present. The 15% masking rate was empirically chosen — too high and the model cannot learn enough about unmasked tokens, too low and pre-training becomes too easy and ineffecive. + +**Q: What are the key architectural differences between BERT (encoder-only) and GPT (decoder-only)?** +A: Four major differences. (1) Attention direction: BERT uses bidirectional (bidirectional) self-attention where every token attends to every other token. GPT uses causal (unidirectional, autoregressive) self-attention where each token can only attend to itself and previous tokens — implemented by masking future tokens in the attention computation. (2) Pre-training objective: BERT uses Masked Language Modeling (predict masked tokens from full context) plus Next Sentence Prediction. GPT uses standard left-to-right language modeling (predict the next token given all previous tokens). (3) Training data: BERT was pre-trained on Wikipedia + BooksCorpus (16 GB text, primarily narrative and factual content). GPT was trained on a broader web corpus (WebText, later Common Crawl, terabytes of text including forums, code, technical documents). (4) Architecture: BERT uses only the Transformer encoder stack (12-24 layers), producing contextualized representations for every input token. GPT uses only the Transformer decoder stack (12-96 layers), which includes masked self-attention and cross-attention to encoder (though in pure decoder-only models, cross-attention is absent). These differences make BERT naturally suited for understanding tasks (classification, QA, NER) where full context is beneficial, while GPT excels at generation (text completion, dialogue, code generation) where left-to-right sequential output is required. + +**Q: Explain the transfer learning approach used with BERT — pre-training and fine-tuning.** +A: BERT uses a two-stage transfer learning approach. Pre-training is the first stage: BERT is trained on a large unlabeled corpus (3.3 billion words) using self-supervised objectives (MLM + NSP). This stage is computationally expensive (4 days on 16 TPUs for BERT-base, $7K+ in cloud compute) but produces a model that understands language deeply — syntax, semantics, world knowledge, and commonsense reasoning are all encoded in the 110M parameters. The pre-trained model is publicly released so others do not need to repeat this expensive step. Fine-tuning is the second stage: the pre-trained BERT model is adapted to a specific downstream task using labeled data. A task-specific classification head (usually 1-2 layers) is added on top. All parameters (pre-trained + new head) are fine-tuned end-to-end, but with a small learning rate (2e-5) to avoid catastrophic forgetting. Fine-tuning is fast: typical tasks converge in 1-10 epochs on a single GPU, often in minutes to hours. The key advantage: BERT transfers general language understanding learned during pre-training, so fine-tuning requires orders of magnitude less labeled data than training from scratch. For example, 10,000 labeled examples fine-tuned on BERT can match or exceed 100,000+ examples trained from scratch. + +### Related Concepts +Transformers, GPT, Fine-tuning, Self-Attention + +## Pre-trained Language Models Comparison +**Definition:** A landscape of major pre-trained language models, each designed with different architectural choices and pre-training objectives that make them suited for different tasks. Understanding these differences is essential for selecting the right model for a given application. + +### Theory & Explanation + +The explosion of pre-trained language models since BERT's introduction in 2018 has created a rich ecosystem of models, each with different strengths. The choice of model depends on the task type (understanding vs generation), computational budget (inference speed, memory), data availability, and deployment constraints. At a high level, models fall into three architectural categories: encoder-only (understanding), decoder-only (generation), and encoder-decoder (sequence-to-sequence). Understanding the core innovation and use case of each major model helps practitioners make informed decisions. + +BERT (Bidirectional Encoder Representations from Transformers) is the foundational encoder-only model. Its key innovation is bidirectional pre-training through masked language modeling, enabling deep contextualized representations for every token. Use cases: classification (sentiment, topic), named entity recognition, question answering (extractive), relation extraction, and any task requiring rich token representations. BERT-base (110M params) and BERT-large (340M params) balance performance and efficiency. The encoder architecture means all tokens are computed simultaneously, making BERT suitable for batch inference on understanding tasks. + +RoBERTa (Robustly Optimized BERT Approach) is a replication study that showed BERT was significantly undertrained. Key innovations: training longer (500K steps vs 100K), larger batches (8K vs 256), more data (160 GB vs 16 GB), dynamic masking (different mask pattern each epoch), and removal of the NSP objective. RoBERTa consistently outperforms BERT on all benchmarks, often by 2-5% on GLUE and SQuAD. Use cases: same as BERT but with higher accuracy. The tradeoff is a longer pre-training process, but since pre-trained models are publicly available, users simply download the better checkpoint. + +DistilBERT is a distilled version of BERT created through knowledge distillation — a smaller student model (teacher is BERT-base) is trained to mimic BERT's output distributions. Key innovations: 40% fewer parameters (66M vs 110M), 60% faster inference, while retaining 95% of BERT's performance. DistilBERT also removes token-type embeddings and the NSP objective. Use cases: production deployment with latency constraints, mobile or edge devices, real-time applications where every millisecond matters, and cost-sensitive environments. Compromise: slightly lower accuracy, particularly on tasks requiring very fine-grained understanding. + +GPT (Generative Pre-trained Transformer) is the foundational decoder-only model. Key innovation: autoregressive (causal) language modeling trained on massive web text. GPT-1 (117M params) showed the potential, GPT-2 (1.5B params) demonstrated zero-shot task transfer, and GPT-3 (175B params) introduced in-context learning. Use cases: text generation, dialogue, code generation, story writing, and any task requiring free-form text output. The autoregressive nature means tokens are generated one at a time, which is slower for generation but enables the model to produce novel text. GPT models also perform understanding tasks through appropriate prompting, though typically less efficiently than encoder-only models. + +T5 (Text-to-Text Transfer Transformer) reframes all NLP tasks as text-to-text problems. Key innovation: a unified framework where every task is expressed as text input → text output. "Translate English to German: The cat" → "Die Katze". "Classify: This movie was great" → "positive". "Summarize: [long article]" → "[summary]". The model is an encoder-decoder Transformer, pre-trained with a denoising objective (masked span prediction). Use cases: translation, summarization, question answering (generative), and any task where task-agnostic architecture is desired. T5's unified approach simplifies deployment (one model for all tasks) at the cost of encoder-decoder inference being slower than encoder-only for understanding tasks. + +BART (Bidirectional and Auto-Regressive Transformer) combines a BERT-like bidirectional encoder with a GPT-like autoregressive decoder. Key innovation: the denoising autoencoder pre-training objective where the input is noised (token masking, text infilling, sentence permutation) and the decoder reconstructs the original text. Use cases: text generation tasks requiring understanding — particularly summarization and dialogue — where the encoder must understand the input while the decoder generates coherent output. BART matches or exceeds T5 on summarization benchmarks and is particularly effective on abstractive summarization tasks that require both understanding and fluent generation. + +XLNet uses permutation language modeling (PLM): instead of masking tokens, it samples all possible orderings of the input tokens and predicts each token in the sampled order. Key innovation: combines the benefits of autoregressive models (no independence assumption) with bidirectional context awareness. XLNet also uses Transformer-XL (relative positional encoding + segment-level recurrence) for handling long sequences. Use cases: tasks where autoregressive modeling helps but bidirectional context is essential — long-document classification, reading comprehension. XLNet often outperforms BERT on benchmarks but is more computationally expensive to train. + +Choosing the right model involves considering: (1) Task type — understanding tasks favor BERT/RoBERTa, generation favors GPT/decoder-only, summarization favors BART/T5. (2) Data size — with limited task-specific data, larger pre-trained models generalize better. (3) Latency requirements — DistilBERT or ALBERT for real-time, GPT-4 or BERT-large for batch. (4) Computational budget — smaller models for edge deployment, larger models for cloud. (5) Sequence length — models with relative positional encoding (RoBERTa, XLNet) handle longer sequences better than absolute position models. (6) Ecosystem and tooling — Hugging Face supports all mentioned models with unified APIs, simplifying experimentation. + +### Example + +Choosing a model for different enterprise NLP tasks. For sentiment analysis on customer reviews (understanding task, 10K labeled examples): BERT-base achieves 92% accuracy, RoBERTa achieves 93.5% but with larger memory, DistilBERT achieves 91% with 60% faster inference. If this is an API serving millions of requests daily, DistilBERT may be the best tradeoff. For text generation in a customer service chatbot: GPT-3.5/4 or Llama 2 (decoder-only) — BERT cannot generate coherent multi-sentence responses. For summarizing quarterly reports (abstractive summarization): BART-large or T5-3B — the encoder-decoder architecture excels at understanding the source document and generating a concise summary. For a question answering system over internal documents (extractive): BERT or RoBERTa fine-tuned on SQuAD format — they find answer spans directly with start/end classifiers. For a code completion tool: GPT-based models (Codex, StarCoder, Code Llama) decoder-only architectures trained on code. For a low-latency on-device NER system: DistilBERT or MobileBERT runs efficiently on smartphone CPUs. If deploying to a serverless function with 512 MB memory: DistilBERT (66M params, ~250 MB) fits; BERT-large (340M params, ~1.3 GB) may exceed the limit. + +### Interview Questions + +**Q: How do you choose the right pre-trained language model for a given task?** +A: The choice depends on five factors. First, task type: understanding tasks (classification, NER, QA) favor encoder-only models like BERT or RoBERTa. Generation tasks (text completion, dialogue, story writing) need decoder-only models like GPT. Sequence-to-sequence tasks (summarization, translation) benefit from encoder-decoder models like BART or T5. Second, accuracy requirements: RoBERTa consistently outperforms BERT across benchmarks (typically 1-3% improvement), but if the accuracy difference is negligible for the specific use case, BERT's smaller size may be preferred. Third, latency and throughput: for real-time applications (<100ms response time), DistilBERT, ALBERT, or quantized versions of larger models are necessary. For batch processing, larger models are acceptable. Fourth, computational budget: consider available GPU memory (BERT-base ~1.2 GB, BERT-large ~3.5 GB, GPT-3 ~350 GB), inference cost, and whether the model runs on cloud or edge devices. Fifth, sequence length requirements: for processing long documents (2000+ tokens), models with relative positional encoding (RoBERTa, XLNet) or efficient attention mechanisms (Longformer, BigBird) are needed. In practice, start with a strong baseline (RoBERTa-base for understanding, GPT-3.5 for generation) and only optimize if needed. + +**Q: What are the key differences between BERT and RoBERTa?** +A: RoBERTa is fundamentally the same architecture as BERT (Transformer encoder, MLM pre-training) but with optimized training procedures. The key differences are: (1) Training data: BERT used 16 GB (Wikipedia + BooksCorpus), RoBERTa increased to 160 GB (adding CommonCrawl News, OpenWebText, and Stories). More data and diverse sources significantly improved language understanding. (2) Training duration: BERT was trained for 100K steps, RoBERTa for 500K steps. With 5x the training, the model learns richer representations. (3) Batch size: BERT used 256 sequences per batch, RoBERTa increased to 8,000. Larger batches stabilize training and improve gradient quality. (4) Dynamic masking: BERT masks tokens once during data preprocessing, so every epoch uses the same mask pattern. RoBERTa applies different random masks for each training example in each epoch, preventing overfitting to specific mask patterns. (5) NSP removal: RoBERTa removed the Next Sentence Prediction objective after experiments showed it was unnecessary or even detrimental. Removing NSP freed capacity for better language modeling. The result: RoBERTa consistently outperforms BERT by 2-5% across GLUE, SQuAD, and RACE benchmarks with zero architectural changes — proving that training optimization matters as much as model design. + +**Q: When would you choose a smaller model like DistilBERT over a full BERT or RoBERTa?** +A: Choose DistilBERT when: (1) inference latency is critical — DistilBERT runs 60% faster than BERT-base, making it suitable for real-time APIs, chatbots, and interactive applications where sub-100ms response times are required. (2) Computational resources are constrained — on mobile devices, edge hardware, serverless functions (512MB memory limit), or when serving many concurrent requests with limited GPU infrastructure, DistilBERT's 66M parameters (vs BERT's 110M) reduces memory and compute requirements significantly. (3) Cost optimization — serving DistilBERT costs roughly half of BERT-base in cloud compute, and when processing millions of requests daily, this adds up substantially. (4) The accuracy drop is acceptable — DistilBERT retains ~95% of BERT's performance, and on many tasks the difference is within 1-2%. For applications where near-perfect accuracy is not strictly required (e.g., preliminary content categorization, social media monitoring), this tradeoff is well worth it. (5) Multiple models need to be deployed — using DistilBERT frees resources for running more models or larger context lengths. The main drawback is on tasks requiring very fine-grained understanding, complex reasoning, or handling of rare linguistic phenomena, where the 5% accuracy gap may matter. + +### Related Concepts +Transformers, BERT, GPT, Fine-tuning + +--- + +# 4. DEEP LEARNING (Advanced) + +## Optimizer +**Definition:** Optimization algorithms that update neural network weights to minimize the loss function by computing gradients and applying parameter updates in the direction that reduces error. + +### Theory & Explanation + +Optimizers control how gradients translate into weight updates. Stochastic Gradient Descent (SGD) is the simplest: it updates weights by subtracting the gradient multiplied by a learning rate — θ = θ - η·∇L(θ). SGD is reliable but converges slowly and can oscillate around local minima. Momentum improves this by accumulating a velocity vector in the direction of persistent gradient, reducing oscillation and accelerating convergence. The update becomes v = β·v + η·∇L(θ), then θ = θ - v. Typical momentum β is 0.9. This works like a ball rolling downhill, gaining speed in consistent directions while dampening oscillations in inconsistent directions. + +Adam (Adaptive Moment Estimation) is the most popular default optimizer, combining Momentum and RMSProp. It maintains two moving averages: the first moment (mean of gradients) and the second moment (uncentered variance of gradients). Adam computes adaptive learning rates for each parameter: m_t = β₁·m_{t-1} + (1-β₁)·g_t and v_t = β₂·v_{t-1} + (1-β₂)·g_t². Default values are β₁=0.9, β₂=0.999, and ε=1e-8. Parameter updates are bias-corrected: θ_t = θ_{t-1} - η·m_t/(√v̂_t + ε). Adam works well out-of-the-box for most architectures, requires little tuning, and handles sparse gradients effectively. + +RMSProp addresses SGD's issue with varying gradient magnitudes across parameters. It divides the learning rate by a running average of squared gradients, effectively slowing updates for parameters with large gradients and accelerating updates for parameters with small gradients. AdaGrad was an earlier adaptive method that accumulates all past squared gradients, but its monotonic learning rate decay causes it to stop learning too early on deep networks. RMSProp's exponential moving average fixes this by giving more weight to recent gradients. + +### Example + +Comparing optimizers on training a CNN for CIFAR-10 (60K 32×32 images, 10 classes). With a fixed learning rate of 0.01, SGD achieves 88% test accuracy after 100 epochs but converges slowly — the loss curve shows gradual improvement with some oscillation near convergence. Momentum (β=0.9) reaches 91% accuracy in 60 epochs, converging faster with smoother loss decay. Adam at the same learning rate hits 92% accuracy in just 40 epochs, though it may not match a well-tuned SGD with momentum on final accuracy. The tradeoff: Adam requires minimal tuning and converges fast, but SGD + momentum with a good learning rate schedule can sometimes generalize better, achieving 93% with careful tuning. + +### Interview Questions + +**Q: When would you choose SGD over Adam?** +A: Choose SGD when: (1) you have enough data and compute to tune the learning rate schedule carefully — SGD with momentum can achieve better final generalization on vision tasks, often beating Adam by 0.5-1% on ImageNet. (2) You need interpretable optimization — SGD's behavior is simpler and more predictable than Adam's adaptive dynamics. (3) You're working with very large batches — Adam's adaptive learning rates can interact poorly with extreme batch sizes, while SGD handles them more robustly. (4) Your model requires sharp minima (some research suggests SGD with large LR finds flatter minima, but carefully tuned SGD can explore both). Choose Adam when you want fast convergence, minimal hyperparameter tuning, or are working with Transformers, RNNs, or sparse data. Most practitioners default to Adam for NLP and Transformers, SGD + Momentum for computer vision. + +**Q: Why does Adam sometimes generalize worse than SGD?** +A: Several hypotheses explain this: (1) Adam's adaptive learning rates can cause the model to converge to sharper minima that generalize poorly — SGD's consistent step size navigates toward flatter minima which typically generalize better. (2) Adam's second moment estimate can become very small for frequently updated parameters, causing large effective learning rates that hurt generalization in later training stages. (3) Adam may overfit to noise in early training because it adapts quickly per-parameter, fitting training set idiosyncrasies rather than general patterns. Solutions include learning rate warmup, using AdamW (decoupled weight decay), or switching to SGD after initial Adam training (Swa or Lookahead optimizers). + +**Q: What is gradient clipping and why is it needed?** +A: Gradient clipping prevents exploding gradients by capping gradient values to a maximum norm threshold, usually set to 1.0 or 5.0. If ||g|| > threshold, gradient is scaled: g = g × threshold / ||g||. This is especially important for RNNs and LSTMs where gradients can grow exponentially through time steps, and for Transformers where attention logits can produce large gradients early in training. Without clipping, a single large gradient update can destabilize the entire network, causing loss to spike to NaN. Clipping ensures stable training by restricting the maximum update magnitude while preserving gradient direction. + +--- + +## Learning Rate +**Definition:** A hyperparameter that controls the step size for updating model weights during gradient descent — how much to adjust the model in response to the estimated error each time weights are updated. + +### Theory & Explanation + +The learning rate (η) is arguably the most important hyperparameter in deep learning. If η is too high, the optimizer overshoots the optimal weights, causing the loss to diverge or oscillate wildly. If η is too low, training progresses extremely slowly, the model may get stuck in poor local minima, and it wastes compute resources. The relationship between learning rate and convergence follows a U-shaped curve: at very low learning rates the model barely moves, at moderate rates it converges smoothly, at high rates it oscillates and may diverge entirely. The optimal learning rate depends on the architecture, dataset, batch size, and optimizer. + +Finding the optimal learning rate is often done with the LR range test (Leslie Smith, 2015): start with a very small learning rate (e.g., 1e-7), exponentially increase it each mini-batch, and track the loss. The optimal LR is where the loss decreases most steeply — typically one order of magnitude below the point where loss starts to increase. For Adam on Transformers, the typical starting point is 1e-4 to 3e-4. For SGD on CNNs, it is 0.01 to 0.1. For fine-tuning large language models, 1e-5 to 5e-5 is standard. + +Learning rate warmup gradually increases the learning rate from near-zero to the target rate over the first few thousand steps. This is critical for Transformers and large batch training because sudden large updates in early training can destabilize the model when weights are randomly initialized and gradient estimates are noisy. A common warmup schedule: linearly increase η from 0 to target over 10% of training steps, then decay for the remaining 90%. + +### Example + +Training ResNet-50 on ImageNet (1.28M images, 1000 classes) with SGD + Momentum. With η=0.1 (standard for ImageNet), training converges to 76% top-1 accuracy in 90 epochs. With η=0.01 (10× too low), after 90 epochs the model only reaches 62% — it learns but much too slowly. With η=0.5 (5× too high), the loss first decreases then oscillates, never converging beyond 40% accuracy — gradient updates keep overshooting the minima. Using cosine annealing from η=0.1 down to 0 over 90 epochs, the model reaches 76.5%, slightly better than constant LR because the decaying rate allows fine-grained convergence in later epochs. + +### Interview Questions + +**Q: Why is warmup particularly important for Transformers?** +A: Transformers have several characteristics that make warmup essential: (1) The Adam optimizer's adaptive learning rates are computed from gradient statistics — early in training, these statistics are extremely noisy because weights are randomly initialized. Without warmup, large early updates based on unreliable statistics can severely destabilize training, causing the model to diverge in the first few hundred steps. (2) LayerNorm's scale and shift parameters (γ, β) are also randomly initialized and need gentle early exploration to find meaningful operating ranges. (3) The multi-head attention mechanism produces large gradient magnitudes when attention patterns are random (uniform attention over many tokens), and a sudden full learning rate can cause gradient explosion through the softmax. Warmup allows the attention patterns to become meaningful before applying full-magnitude updates. + +**Q: What is the relationship between learning rate and batch size?** +A: There is a well-established scaling rule: when doubling the batch size, the learning rate can be approximately doubled (linear scaling rule). Larger batches produce lower-variance gradient estimates, allowing larger steps. With batch size B, gradient variance scales as 1/B, so the safe LR scales linearly with B up to a point. However, this relationship breaks at very large batch sizes (e.g., >4096) because diminishing returns in variance reduction mean further LR increases cause instability. The practical guideline: if you increase batch size from 256 to 512, increase LR by ~1.5-2×, but above 4096, scale more conservatively (e.g., 1.2-1.5×) and consider additional warmup steps. + +--- + +## Epoch & Batch +**Definition:** An epoch is one complete pass of the entire training dataset through the neural network; a batch is a subset of training examples processed together in one forward/backward pass. + +### Theory & Explanation + +The distinction between epoch, batch, and iteration is fundamental to understanding training dynamics. An epoch represents one full traversal of the training data — every example has been seen exactly once by the model. If the dataset has N examples and the batch size is B, each epoch contains ceil(N/B) iterations (parameter updates). Training typically requires multiple epochs (10-100+) for the model to converge, as each epoch provides a fresh pass through the data with different batch compositions (assuming shuffling). The total number of parameter updates is epochs × (N/B). + +Mini-batch gradient descent is the standard approach: the dataset is split into small batches of 32-512 examples. This balances two competing concerns: computational efficiency and gradient accuracy. Full-batch gradient descent (B=N) gives the true gradient but is computationally expensive for modern datasets (millions of examples). Stochastic gradient descent (B=1) updates parameters after every example, producing noisy gradients that oscillate heavily but can sometimes escape poor local minima. Mini-batch provides a middle ground — noisy enough to regularize but accurate enough for stable convergence. + +Batch size has a significant impact on generalization. Research has shown that large batch sizes tend to converge to sharper minima that generalize poorly, while small batch sizes find flatter minima with better test performance. This is because small-batch gradients contain more noise, which acts as a regularizer and helps the optimizer escape sharp minima. The practical implication: if your model is overfitting, try smaller batches; if your model generalizes well but training is slow, try larger batches. Modern hardware accelerators (GPUs/TPUs) are most efficient with batch sizes that are powers of 2 (32, 64, 128, 256, 512) due to memory alignment. + +### Example + +Training BERT-base (110M parameters) on Wikipedia + BooksCorpus (3.3B tokens). With batch size 256 and sequence length 512, each iteration processes 256 × 512 = 131,072 tokens. The total training data is about 3.3B tokens, so one epoch = 3.3B/131,072 ≈ 25,200 iterations. BERT was trained for 1M iterations, which is 1,000,000/25,200 ≈ 40 epochs. At the time, this took 4 days on 16 TPUs. With a batch size of 512 (doubled), each epoch would take half the iterations (12,600), potentially reducing training time to 2 days if the LR is scaled appropriately. However, the larger batch might need 50 epochs to generalize equivalently, requiring a total training time of ~2.5 days. + +### Interview Questions + +**Q: Why does a large batch size sometimes hurt generalization?** +A: Several reasons: (1) Large batches reduce gradient noise — noise during training acts as a regularizer that helps the optimizer escape sharp minima and explore flatter regions of the loss landscape. With noise reduced, the model converges to sharp minima that have high test error. (2) Large batches require fewer parameter updates overall — fewer updates mean the model explores less of the parameter space before converging, missing potentially better solutions. (3) Large batch training tends to overspecialize — with fewer but more accurate gradient steps, the model fits the training data more precisely without the exploration that noise provides. Solutions: increase training epochs proportionally, use learning rate warmup and careful LR scaling, or use adaptive batch size methods (Ghost Batch Normalization, gradual batch size increase). + +**Q: How do you choose the optimal batch size?** +A: Optimal batch size depends on several factors: (1) Hardware constraints — GPUs have limited memory; the maximum batch size is determined by how many examples fit in VRAM. Typical values: 32-128 for large models (BERT, GPT), 256-1024 for medium models (ResNet-50), up to 8192 for small models with large GPU clusters. (2) Convergence requirements — if the model converges within 1-2 epochs (rare), batch size matters less. For most deep learning, 32-512 works well. (3) Training efficiency — measure throughput (examples/second) across batch sizes. The optimal batch size is usually where throughput saturates (before diminishing returns from parallelization). (4) Generalization needs — smaller batches for regularization (start at 32 and increase if training is stable but slow). Rule of thumb: start with 32, double until throughput per GPU stops improving, then keep that size. + +--- + +## Gradient Descent Variants +**Definition:** Variations of the gradient descent algorithm that differ in how much data is used to compute the gradient at each update step — full dataset (Batch GD), one sample (Stochastic GD), or a subset (Mini-batch GD). + +### Theory & Explanation + +Batch Gradient Descent computes the gradient of the loss function over the entire training dataset before making a single weight update. The update rule is θ = θ - η × (1/N) × Σ∇L(x_i, y_i, θ). This produces the true gradient direction, guaranteeing monotonic decrease of the loss function (for convex problems) and stable convergence. However, for modern datasets with millions of examples, computing the full gradient is computationally prohibitive — each update requires processing the entire dataset, and the update frequency is once per epoch. For a dataset of 1M images, Batch GD makes only ~100 updates in 100 epochs, which is far too few for convergence. + +Stochastic Gradient Descent (SGD) goes to the opposite extreme: it updates weights after every single training example. The update is θ = θ - η × ∇L(x_i, y_i, θ) for one random example i. This introduces high variance in gradient estimates — each individual example's gradient can point in a very different direction from the true gradient. The high noise means convergence is not smooth; the loss bounces up and down even as it trends downward. However, this noise can be beneficial: it helps the optimizer escape shallow local minima and saddle points that would trap batch GD. SGD requires many more iterations (one per example vs one per epoch) but each iteration is very fast. + +Mini-batch Gradient Descent is the practical compromise used in virtually all modern deep learning. It uses a subset of B examples, computing θ = θ - η × (1/B) × Σ∇L(x_j, y_j, θ). Batch sizes of 32-512 balance gradient accuracy with computational efficiency. The gradient estimate from a mini-batch is noisy enough to escape poor minima but accurate enough for stable convergence. Mini-batch also enables efficient GPU parallelization — modern GPUs process B examples simultaneously with minimal overhead compared to processing a single example, making B=1 (pure SGD) counterintuitively slower per-epoch than B=64. + +### Example + +Training a logistic regression model on 10,000 examples with 20 features. Batch GD computes one gradient over all 10,000 examples, costing 10,000 × 20 = 200K operations per update. After 1,000 epochs, it makes 1,000 updates at total cost 200M operations. Pure SGD computes 10,000 updates per epoch, each costing 1 × 20 = 20 operations. After 1,000 epochs, it makes 10M updates at total cost 200M operations — same total compute but vastly different convergence dynamics. SGD converges rapidly in the first few epochs (each update reduces loss quickly) but spends the remaining epochs oscillating around the minimum. Mini-batch with B=64 makes 156 updates per epoch (10,000/64), each costing 64×20=1,280 operations. After 1,000 epochs: 156K updates at 200M total operations, converging faster than batch GD and more stably than pure SGD. + +### Interview Questions + +**Q: Why is mini-batch GD the default choice in practice?** +A: Three key reasons: (1) Gradient quality vs computation tradeoff — mini-batch provides a gradient estimate with variance B times lower than pure SGD but at B times the per-update cost. The sweet spot (32-512) gives reliable gradient estimates with efficient hardware utilization. (2) GPU parallelism — modern GPUs process batches efficiently; a batch of 64 examples takes roughly the same time as 1 example due to SIMD parallelism, making pure SGD 64× slower per-epoch for the same number of gradient computations. (3) Regularization effect — the noise from mini-batch estimation (compared to full-batch) serves as implicit regularization, helping generalization. Too much noise (SGD) or too little (batch GD) both hurt — mini-batch provides the right amount. + +**Q: What happens to convergence as batch size approaches 1 or N?** +A: As batch size approaches 1 (pure SGD): convergence becomes very noisy — the loss oscillates significantly, requiring a learning rate schedule (typically decaying) to converge. The model may find better minima due to the noise but takes many more iterations. As batch size approaches N (full batch): convergence is smooth and deterministic but extremely slow per-update. The model gets stuck in poor local minima and saddle points more easily. The noise vanishes, meaning no implicit regularization. The practical implication: use batch sizes between 32 and 512 for most tasks; only use full batch for very small datasets (N < 1000) or when exact gradients are needed for debugging. + +--- + +## Dropout +**Definition:** A regularization technique that randomly drops (sets to zero) a fraction of neurons during training, preventing co-adaptation by forcing the network to learn redundant representations. + +### Theory & Explanation + +Dropout works by probabilistically removing neurons during each forward pass. For a given layer with dropout rate p (typically 0.2 to 0.5), each neuron is independently kept active with probability 1-p and dropped with probability p during training. The dropped neurons do not participate in forward propagation or backward propagation — their contribution to the output and their weight updates are temporarily zeroed out. This means each training iteration effectively trains a different "thinned" subnetwork that shares weights with the full network. At inference time, dropout is disabled and all neurons are active, with their outputs scaled by (1-p) to maintain consistent expected activation magnitude. + +The regularization effect of dropout comes from preventing co-adaptation among neurons. Without dropout, neurons can learn to rely on specific other neurons to correct their errors — they develop interdependent features. Dropout breaks these co-dependencies by randomly removing neurons, forcing each neuron to learn robust, useful features independently. The result is that the network learns redundant representations: if one feature detector is dropped, others can pick up the slack. This is analogous to ensemble learning: dropout trains an ensemble of exponentially many subnetworks (2^n for n neurons) that share weights, and inference averages their predictions. + +The standard implementation is inverted dropout. During training, kept neuron outputs are divided by (1-p) to maintain the same expected total activation. This means h_train = (mask × h) / (1-p), where mask is a binary vector with probability (1-p) of being 1. During inference, no scaling is needed — all neurons are active at full strength. Inverted dropout is preferred because it requires no inference-time modification to the model, making deployment simpler and faster. Modern frameworks (PyTorch, TensorFlow) implement inverted dropout by default. + +### Example + +Training a 2-layer MLP (512 → 256 → 10) on MNIST (60K handwritten digits, 10 classes). Without dropout, the model achieves 99.2% training accuracy but 97.5% test accuracy after 50 epochs — a 1.7% generalization gap indicating overfitting. Adding dropout (p=0.5) to the hidden layers reduces training accuracy to 97.8% but maintains 98.3% test accuracy — the gap shrinks from 1.7% to -0.5% (test accuracy higher than training, common with strong regularization). With p=0.2, the model achieves 98.8% training and 98.2% test accuracy — less regularization allows higher training accuracy while still improving generalization over no dropout. The optimal dropout rate depends on overfitting severity: heavy overfitting → higher p, mild overfitting → lower p. + +### Interview Questions + +**Q: Why does inverted dropout scale by 1/(1-p) during training instead of scaling by (1-p) at test time?** +A: Inverted dropout scales during training rather than inference for practical deployment reasons. The scaling at training time (dividing kept activations by 1-p) ensures that the expected sum of activations at each layer is the same regardless of the dropout rate. At inference, all neurons are active, and no scaling is needed because each neuron's output already reflects its true contribution. This means inference code is identical to a model without dropout — the model can be shipped and served without any special handling. If we instead scaled at test time (multiplying by 1-p), we would need to track which layers had dropout and modify the inference code accordingly, increasing complexity and the risk of forgetting to apply scaling. Inverted dropout makes deployment cleaner: training handles all the complexity, inference is standard forward pass. + +**Q: How does dropout interact with batch normalization?** +A: Dropout and batch normalization can interact poorly if not handled carefully. Batch normalization computes its statistics (mean and variance) over the current batch, and dropout creates different neuron activation patterns in each iteration. This causes two issues: (1) The variance of BN statistics increases because dropout is randomly zeroing activations, making the normalization less stable. (2) Dropout reduces the effective batch size for BN's statistics computation, increasing estimation noise. The common recommendation is to use either dropout OR batch normalization, not both in the same layer. When combined, use a lower dropout rate (p=0.1-0.2). Some architectures avoid dropout entirely by relying on BN's built-in regularization noise. In modern practice, dropout is more commonly used in fully connected layers (Transformers, MLP heads) while CNNs rely more on BN for regularization. + +**Q: What is the connection between dropout and ensemble methods?** +A: Dropout is essentially efficient ensemble learning. With n neurons, there are 2^n possible subnetworks (each neuron either present or absent). Each training iteration trains a different random subnetwork. At test time, all subnetworks share weights through the dropout mechanism, and using all neurons approximates taking the geometric mean of all 2^n ensemble members' predictions. This is why dropout is so effective: it gives the benefits of a massive ensemble without the computational cost of training and storing 2^n separate models. The key insight is weight sharing across ensemble members, which makes the ensemble computationally feasible and provides strong regularization. + +--- + +## Batch Normalization +**Definition:** A technique that normalizes layer inputs to have zero mean and unit variance across the batch, then applies learnable scale (γ) and shift (β) parameters to restore representational power. + +### Theory & Explanation + +Batch Normalization (BN) operates on each mini-batch during training. For each feature dimension, it computes the batch mean μ_B = (1/m) × Σx_i and batch variance σ²_B = (1/m) × Σ(x_i - μ_B)², then normalizes: x̂_i = (x_i - μ_B) / √(σ²_B + ε), where ε is a small constant (1e-5) to prevent division by zero. The normalized values have mean 0 and variance 1, ensuring inputs to activation functions remain in regions with non-zero gradients. Finally, BN applies a learnable linear transformation: y_i = γ × x̂_i + β, where γ (scale) and β (shift) are learned during training via backpropagation. + +BN addresses a problem called internal covariate shift — the distribution of layer inputs changes as earlier layer weights are updated during training. This shift forces later layers to continuously adapt to changing input distributions, slowing convergence. BN stabilizes the distribution of each layer's inputs, allowing higher learning rates (typically 5-10× larger) because the normalized activations prevent extreme updates from amplifying through the network. The normalization also prevents activations from saturating in the flat regions of sigmoid/tanh activations, alleviating the vanishing gradient problem. + +During training, BN uses batch statistics, which introduce a small amount of noise (from random batch composition). This noise acts as a regularizer, similar to dropout — each example's representation is slightly different depending on which other examples are in its batch. At inference time, BN switches to using running averages of μ and σ² accumulated during training (often computed with exponential moving average, momentum typically 0.9 or 0.99). This makes inference deterministic — predictions depend only on the input, not on which other examples happen to be in the batch. The running estimates are maintained throughout training and frozen for inference. + +### Example + +Consider a deep CNN trained on ImageNet with ReLU activations. Without BN, a learning rate of 0.01 causes gradients to vanish in early layers (first few convolutional layers receive gradients ~1e-4 × final layer gradient). The model reaches 50% top-1 accuracy after 50 epochs. Adding BN with the same learning rate allows training at 0.1 (10× higher), and the model reaches 73% top-1 accuracy in 50 epochs — the normalization prevents gradient attenuation through depth. With BN, training accuracy reaches 99% within 30 epochs because the optimizer can take much larger steps. The γ and β parameters, which start as γ=1, β=0, evolve during training: for the last convolutional layer, γ settles to 1.3 and β to -0.2, indicating the layer slightly amplifies and shifts the normalized distribution for optimal downstream performance. + +### Interview Questions + +**Q: Why does batch normalization allow higher learning rates?** +A: BN prevents small changes in early layers from being amplified into large changes in later layers. Without normalization, a small update to early layer weights changes the distribution of their outputs, which propagates through subsequent layers and is amplified — a small change at layer 1 can produce a large change at layer 10. This means the effective learning rate for later layers is much higher than set, causing instability. BN normalizes each layer's inputs to the same scale regardless of earlier layer changes, decoupling layers. This enables 5-10× higher learning rates because the mapping from weight update to output change is stable and predictable across all layers. + +**Q: What are the downsides of batch normalization?** +A: Several limitations: (1) Batch size dependency — BN performs poorly with small batches (e.g., batch size 1-8 for medical images or video). The batch statistics become noisy estimates of the true population statistics, adding harmful noise rather than useful regularization. (2) Different training and inference behavior — BN uses batch statistics during training and running averages during inference, creating a discrepancy that can cause model performance to drop when deploying batch size 1 (common in production). This discrepancy is especially pronounced when training and inference batch distributions differ. (3) Sequence models — for RNNs/LSTMs, BN is difficult to apply correctly because sequence length varies and batch statistics across time steps are problematic. LayerNorm is typically preferred for sequential models. (4) Overhead — BN adds computation, memory (storing running statistics), and complexity to the training pipeline. + +**Q: How do running mean and variance work during inference?** +A: During training, BN maintains exponential moving averages of batch mean and variance: μ_running = momentum × μ_running + (1-momentum) × μ_batch, and similarly for variance. The momentum is typically 0.9 or 0.99, meaning the running average smooths over roughly 10/100 recent batches. These running estimates are updated at each training step but are frozen at the end of training. During inference, the BN layer uses these fixed running statistics: y = γ × (x - μ_running) / √(σ²_running + ε) + β. This ensures deterministic output regardless of batch composition. In practice, frameworks like PyTorch and TensorFlow handle this automatically when switching between train and eval modes. + +--- + +## Layer Normalization +**Definition:** A normalization technique that computes mean and variance across the feature dimension (hidden units) rather than the batch dimension, enabling stable training regardless of batch size. + +### Theory & Explanation + +Layer Normalization (LN) normalizes the activations for each individual training example independently, computing statistics across all neurons in a layer. For a layer with H hidden units, LN computes μ = (1/H) × Σx_i and σ² = (1/H) × Σ(x_i - μ)², then normalizes: x̂_i = (x_i - μ) / √(σ² + ε). Like BN, it applies learnable scale γ and shift β: y_i = γ × x̂_i + β. The critical difference is that LN normalizes across features for each sample, while BN normalizes across samples for each feature. This means LN's behavior is identical during training and inference — no running statistics, no batch dependency. + +LN is the preferred normalization for Transformers and RNNs for several reasons. In Transformers, the self-attention mechanism operates on all positions simultaneously, and the effective "batch size" in the batch dimension can be small or variable. LN's independence from batch composition makes it naturally suited. For RNNs, where sequence length varies and the recurrent state evolves through time, BN is problematic because the normalized statistics change unpredictably across time steps. LN normalizes the entire hidden state at each time step, stabilizing the recurrent dynamics. Consequently, virtually all modern Transformer architectures (BERT, GPT, T5) and LSTM variants use LayerNorm. + +LN also behaves differently under distribution shift. Since LN computes statistics per-sample, the normalization adapts to the specific scale of each example. For NLP tasks where input sequence lengths vary, LN ensures that short and long sequences have similar activation distributions. In vision tasks, LN has been adopted in Vision Transformers (ViT) and has been explored as an alternative to BN in CNNs. Group Normalization (GN) is a generalization that splits features into groups and normalizes within each group — it performs between LN (one group) and BN (H groups of size 1) and is useful for small batch sizes in vision. + +### Example + +Consider a BERT-base Transformer with 12 layers, each having a hidden size of 768. Each attention sublayer and feedforward sublayer applies LayerNorm with γ initialized to 1 and β initialized to 0. During training, for an input sequence of length 128: each of the 128 positions has its own 768-dimensional vector, and LN computes 128 separate sets of μ and σ² — one per position, each over 768 features. During training, γ values in the later layers converge to ~0.8-1.2 and β to ~-0.1 to 0.1. If we used BN instead with a batch size of 32, BN would compute 768 separate statistics — one per feature dimension across 32 × 128 = 4096 values. This works fine with batch size 32 but behaves unreliably at inference when batch size might be 1 (single example inference). + +### Interview Questions + +**Q: Why is LayerNorm preferred over BatchNorm for Transformers?** +A: Three fundamental reasons: (1) BN's behavior depends on batch composition — for NLP tasks where batch size varies (different numbers of sequences or variable-length sequences), BN's batch statistics are inconsistent. LN processes each example independently, giving consistent output regardless of batch composition. (2) BN introduces dependence between examples in the batch — the representation of one token depends on which other examples are in its batch. For autoregressive generation (like GPT), this is undesirable because generation typically processes one sequence at a time, and training should match inference behavior. (3) BN's running statistics create a mismatch between train and inference — during training at batch size 64, BN computes statistics that depend on batch composition; during inference, the frozen running statistics may not match the actual data distribution. LN eliminates this mismatch entirely by using per-example statistics at both train and inference time. + +**Q: What happens if you apply BatchNorm to a Transformer?** +A: Applying BN to a Transformer causes several issues: (1) Training instability — BN's batch statistics vary significantly across mini-batches in NLP because the data distribution varies per example (unlike vision where images share similar pixel distributions). This variance causes larger gradient noise and higher loss spikes. (2) Poor performance with variable-length sequences — padding tokens introduce all-zero distributions that skew batch statistics. Even with padding masks, BN's synchronized computation across positions creates artifacts. (3) Sequence modeling problems — for decoder-only Transformers (GPT style) that generate tokens autoregressively, BN would need to compute statistics over future tokens during training, which is impossible without cheating (using future tokens to normalize current tokens). (4) Empirical evidence — experiments where BN replaces LN consistently show 3-5% worse perplexity on language modeling benchmarks and slower convergence, even with careful tuning of BN momentum and epsilon. + +--- + +## Weight Initialization +**Definition:** The process of setting initial values for neural network weights before training begins, critical for enabling effective gradient flow and preventing vanishing or exploding gradients in deep networks. + +### Theory & Explanation + +Weight initialization is crucial because gradient magnitudes depend on the product of weight values through the network. Consider a forward pass through L layers: the output of layer l is x_l = W_l × a_{l-1}, where a_{l-1} = σ(x_{l-1}). During backpropagation, the gradient at layer l is proportional to the product of weights from layers l+1 to L. If weights are too large, this product grows exponentially with depth (exploding gradients). If weights are too small, it decays exponentially (vanishing gradients). Good initialization aims to keep the variance of activations and gradients constant across layers, achieving the so-called "Glorot condition." + +Xavier/Glorot Initialization (2010) was designed for tanh and sigmoid activations. It initializes weights from a uniform distribution: W ~ U[-√(6/(n_in + n_out)), √(6/(n_in + n_out))] or from a normal distribution: W ~ N(0, √(2/(n_in + n_out))). The key insight is that to keep the variance of activations constant, the initial weight variance should be 2/(n_in + n_out). For tanh activations (which are approximately linear near 0), this ensures that the forward pass doesn't amplify or attenuate signal strength. For sigmoid, the same works, but sigmoid's saturating behavior makes it more sensitive to initialization. + +He Initialization (Kaiming He, 2015) was developed specifically for ReLU activations. Since ReLU sets half the activations to zero (negative values), the effective variance is halved. He initialization compensates by increasing the initial variance: W ~ N(0, √(2/n_in)). This ensures that through ReLU layers, the forward pass preserves variance. In practice, models using ReLU with Xavier initialization often experience diminishing gradients in early layers — the variance shrinks through the network. He initialization fixes this. For modern architectures (ResNet, EfficientNet, Vision Transformer), He initialization is standard. Leaky ReLU and PReLU variants use the same formula with slight modifications to account for the negative slope. + +### Example + +Training a 50-layer MLP (1000 → 1000 → ... → 10) with ReLU activations on CIFAR-100. With small random initialization (W ~ N(0, 0.01)), after forward pass of the first batch, activations in layer 50 have variance ~1e-15 — effectively all zeros. The network cannot learn (training accuracy stays at 1%). With Xavier/Glorot (W ~ N(0, 1/500)), activations at layer 50 have variance ~1e-5 — still nearly dead. With He initialization (W ~ N(0, 2/1000)), activations at layer 50 have variance ~0.9 — signal propagates well. The model reaches 50% test accuracy within 100 epochs. If we use tanh activations instead, He initialization over-amplifies variance (activations saturate at ±1), while Xavier keeps variance stable at ~1.0, achieving 48% test accuracy. + +### Interview Questions + +**Q: Why does wrong initialization cause vanishing or exploding gradients?** +A: Consider a simple network with L layers, each with weight matrix W_l and ReLU activation. During forward pass, the output variance approximately multiplies by Var[W] × 0.5 (from ReLU) each layer. After 100 layers, variance scales by (Var[W] × 0.5)^100. If Var[W] = 0.01, this becomes (0.005)^100 ≈ 10^-230 — activations vanish to zero. During backward pass, the gradient variance also multiplies layer by layer. If Var[W] = 2, then (2 × 0.5)^100 = 1^100 = 1 — stable. But if Var[W] > 2 (e.g., 3), the product exceeds 1 and gradients explode exponentially. The mathematical condition: to avoid vanishing or exploding, the weight variance should be approximately 2/n_in (He) or 2/(n_in + n_out) (Xavier) to keep the product of variances through layers stable. + +**Q: What initialization is recommended for different activation functions?** +A: For sigmoid/tanh: Xavier/Glorot initialization. For ReLU and its variants (Leaky ReLU, PReLU): He (Kaiming) initialization. For ELU/SELU: LeCun initialization (W ~ N(0, 1/n_in)) — SELU specifically requires it for self-normalizing properties. For linear activations: Xavier is fine. In practice, most modern architectures use He initialization with ReLU by default. For Transformers, default initialization varies: PyTorch's nn.Transformer uses truncated normal with std=0.02, while TensorFlow's BERT uses truncated normal with std=0.02 for weights and 0.01 for embeddings. The key principle: initialize such that the variance of activations and gradients remains approximately 1 through the network depth. + +--- + +## Learning Rate Scheduler +**Definition:** A strategy to adjust the learning rate during training, typically starting higher for fast progress and decreasing to fine-tune convergence. + +### Theory & Explanation + +Learning rate schedulers are essential because the optimal step size changes over the course of training. Early in training, weights are far from optimal and larger steps enable rapid progress through the loss landscape. Later, as the model approaches a minimum, smaller steps are needed to fine-tune without overshooting. Without scheduling, a fixed learning rate either stays too large (oscillation without convergence) or too small (extremely slow early progress). Modern schedulers systematically decrease the learning rate according to a predefined plan or based on training dynamics. + +Step decay reduces the learning rate by a constant factor (typically 0.1) every fixed number of epochs. Example: train ResNet-50 for 90 epochs, reduce LR by 10× at epochs 30 and 60. This is simple and effective but requires domain knowledge to set the decay points. Exponential decay applies a continuous decay: η_t = η_0 × e^{-kt}, where k is the decay rate. This is smooth but may decay too quickly early on or too slowly later. Cosine annealing follows a cosine curve: η_t = η_min + 0.5 × (η_max - η_min) × (1 + cos(t/T × π)), where T is the total number of iterations. It decays slowly at first, then faster in the middle, then slowly again, providing a good balance. + +ReduceLROnPlateau is an adaptive method that reduces the learning rate by a factor (typically 0.5-0.2) when a monitored metric (validation loss, accuracy) stops improving for a set number of epochs (patience, typically 5-10). This is extremely practical because it automatically detects when the model has reached a plateau, avoiding manual schedule tuning. OneCycleLR (Leslie Smith, 2018) is a modern approach that first linearly increases the LR from a low value to a maximum, then linearly decreases to near-zero. The "warmup + annealing" cycle completes in one cycle of training, hence the name. It allows very high maximum learning rates and often achieves better results in fewer epochs. + +### Example + +Training ResNet-50 on ImageNet for 90 epochs. With a fixed LR of 0.1, the model reaches 73% top-1 accuracy. With step decay (0.1 → 0.01 at epoch 30 → 0.001 at epoch 60), accuracy reaches 76.1% — the model makes fast progress first, then fine-tunes. With cosine annealing (η_max=0.1, η_min=0), accuracy reaches 76.5%. With ReduceLROnPlateau (factor=0.1, patience=3, monitor=val_loss), the LR drops from 0.1 to 0.01 around epoch 25, then to 0.001 around epoch 50, achieving 76.3% — slightly worse than scheduled but requiring zero manual tuning of decay points, making it ideal for new domains. OneCycleLR (max_lr=0.1, pct_start=0.3) reaches 77.1% in just 50 epochs — the high max LR enables faster progress, and the cycle ends at near-zero LR for precise final convergence. + +### Interview Questions + +**Q: Why does cosine annealing often outperform step decay?** +A: Cosine annealing provides a smooth, gradual decay that adapts to different training phases: (1) In early training, the cosine curve decays slowly, allowing the model to explore the loss landscape with large steps. (2) In mid-training, the decay accelerates, transitioning from exploration to refinement. (3) In late training, the decay slows again, enabling very fine-grained convergence. Step decay abruptly changes the LR by orders of magnitude, which can shock the optimizer — the model must adapt to a suddenly much smaller step size, potentially wasting epochs. Cosine scheduling eliminates these discontinuities, allowing 0.5-2% improvements on various benchmarks, especially with longer training schedules. + +**Q: How do you warmup learning rates and why does it help?** +A: Warmup gradually increases the learning rate from a very small value (near 0) to the target value over the first few thousand steps (typically 5-10% of total training). This helps because: (1) Early training has extremely noisy gradient estimates — weights are randomly initialized, and the gradient direction for the first few batches is largely determined by initialization artifacts rather than meaningful data patterns. Taking large steps with these noisy gradients can push weights into bad regions of the loss landscape that are hard to escape. (2) For Adam, the first and second moment estimates need time to accumulate before they provide reliable adaptive rates — warmup provides this buffer. (3) For Transformers specifically, the attention mechanism produces very sharp gradients initially because attention patterns are uniform over all tokens — a full LR can cause gradient explosion through the softmax. Common warmup: linear increase from 0 to η_max over the first 10% of training, often paired with cosine decay for the remaining 90%. + +--- + +## Data Augmentation +**Definition:** The technique of creating modified versions of training data to artificially expand the dataset size and diversity, improving model generalization and reducing overfitting. + +### Theory & Explanation + +Data augmentation works by applying label-preserving transformations to existing training examples, generating new samples that the model sees as distinct from the originals. This increases the effective size and diversity of the training set without collecting new data. For image classification, a rotation of a "cat" image is still a cat — the model learns to be invariant to orientation. Each augmentation type teaches the model a specific invariance: horizontal flips teach mirror invariance, color jitter teaches lighting invariance, random crops teach translation invariance. The key is choosing augmentations that do not alter the label — a 90-degree rotation of a "6" digit becomes a "9", which would be misleading. + +Common image augmentations include geometric transforms (rotation: ±30°, horizontal flip: 50% probability, random crop: 10-20% of image size, translation, scaling, shearing) and photometric transforms (brightness: ±0.2, contrast: ±0.2, saturation: ±0.2, hue: ±0.1, Gaussian noise, blur). Cutout randomly masks a square region (8-16% of image), forcing the model to rely on context rather than a single discriminative feature. MixUp creates convex combinations of pairs of examples and their labels: x̃ = λx_i + (1-λ)x_j, ỹ = λy_i + (1-λ)y_j, where λ ~ Beta(α, α) typically with α=0.2-0.4. This encourages linear behavior between training examples and improves robustness. + +For NLP, augmentation is more challenging because discrete text is harder to transform while preserving meaning. Common techniques include synonym replacement (replace words with synonyms using WordNet or contextual embeddings), random deletion or insertion, random swap of adjacent words, and back-translation (translate text to another language and back, preserving meaning while altering surface form). For text classification, EDA (Easy Data Augmentation) combines these techniques with low probabilities per word (typically 10-20%). Back-translation is particularly effective because it generates fluent, semantically equivalent text with different lexical choices. More advanced approaches use BERT or GPT to generate realistic augmentations in a controlled manner. + +### Example + +Training a ResNet-50 on CIFAR-100 (50K training images, 100 classes). Without augmentation, the model reaches 100% training accuracy (overfitting) but only 65% test accuracy — a 35% gap. Adding random horizontal flip (±50% probability) and random crop (32×32 → 32×32 with 4px padding) reduces training accuracy to 95% and increases test accuracy to 75%. Adding Cutout (16×16 mask) and color jitter (brightness=0.2, contrast=0.2) further improves test accuracy to 78%. Adding MixUp (α=0.2) pushes test accuracy to 82% while training accuracy stays at 90% — the interpolated labels prevent exact memorization. Total improvement from augmentation: 65% → 82%, a 17% absolute gain without any new data collection. + +### Interview Questions + +**Q: Why does MixUp improve model generalization?** +A: MixUp improves generalization through several mechanisms: (1) It creates a linear interpolation between training examples, encouraging the model to have a linear decision boundary between classes rather than sharp, complex boundaries that memorize individual examples. This linearity typically generalizes better. (2) It exposes the model to examples that lie in the convex hull of training data — these "virtual" examples fill gaps in the training distribution, making the model more robust to out-of-distribution inputs near training examples. (3) MixUp with blended labels prevents the model from becoming overconfident in its predictions — the model learns that intermediate inputs correspond to intermediate labels, resulting in better calibrated probabilities. (4) It acts as a strong regularizer without requiring explicit penalty terms, reducing overfitting by 3-15% on various benchmarks. + +**Q: When does data augmentation fail or hurt performance?** +A: Data augmentation can hurt in several scenarios: (1) Label-destroying augmentations — applying a 90° rotation to MNIST digit "6" turns it into "9", teaching the model wrong labels. Any augmentation that changes the class identity is harmful. (2) Excessive augmentation strength — extreme color jitter that makes images unrecognizable (e.g., hue shift of ±0.5) destroys useful information. The optimal augmentation strength is often dataset-dependent. (3) Insufficient data — with very small datasets, even well-designed augmentation may not compensate for lack of real diversity. (4) Mismatched test distribution — training with augmentations that don't reflect real-world variations. For instance, extreme rotations on a license plate recognition task hurt because test images are always upright. (5) Computational overhead — heavy on-the-fly augmentation can significantly slow training. The solution is to test augmentation policies on a validation set and use techniques like AutoAugment or RandAugment to learn optimal policies. + +--- + +## Autoencoder +**Definition:** An unsupervised neural network trained to reconstruct its input through a bottleneck layer, learning a compressed latent representation that captures the most salient features of the data. + +### Theory & Explanation + +An autoencoder consists of two components: an encoder that maps the input to a compressed latent representation, and a decoder that reconstructs the original input from this latent code. The architecture forces the network to learn an efficient encoding by creating a bottleneck — the latent representation has lower dimensionality than the input. During training, the model minimizes the reconstruction error (typically MSE for continuous data, binary cross-entropy for images), learning which features are important enough to preserve through the bottleneck. The loss is simply L(x, x̂) = ||x - x̂||² or similar, requiring no labels — making autoencoders purely unsupervised. + +The latent representation (bottleneck) is the key output. For a 784-pixel MNIST image compressed to 32 dimensions, the autoencoder must learn to capture the essential structure: digit type, stroke width, slant, and other discriminative features while discarding pixel-level noise. The encoder's weights learn to extract these features automatically through reconstruction pressure. Undercomplete autoencoders (smaller bottleneck) learn the most salient features. Overcomplete autoencoders (larger bottleneck) need additional regularization (sparsity, denoising) to prevent simply copying input to output without learning meaningful representations. + +Variations of autoencoders include Denoising Autoencoders (DAE), which corrupt the input with noise and train the model to reconstruct the clean version — this forces the model to learn robust feature extraction that ignores irrelevant noise. Sparse Autoencoders add a sparsity penalty on latent activations, forcing the model to use only a subset of latent neurons for each input. Variational Autoencoders (VAE) are a probabilistic variant that learns a distribution over the latent space, enabling generative capabilities — they can sample from the learned latent distribution to generate new data. Contractive Autoencoders add a penalty on the Jacobian of the encoder to learn features that are locally invariant. + +### Example + +Training a Denoising Autoencoder on MNIST. Encoder: 784 → 256 → 64 → 32 (ReLU). Decoder: 32 → 64 → 256 → 784 (sigmoid). Input images are corrupted with Gaussian noise (std=0.3). After 50 epochs, the model achieves MSE of 0.008 on clean test images. Original noisy digits are nearly unrecognizable (e.g., a "7" with noise appears as gray blobs), but the reconstructed output is a clean "7" — the autoencoder has learned the manifold of handwritten digits. The 32-dimensional latent vectors cluster by digit class (visually separable with t-SNE), demonstrating that the autoencoder learned semantically meaningful features without any labels. For anomaly detection: a "normal" digit (0-9) reconstructs with MSE < 0.01, but an anomalous input (e.g., a random noise pattern) reconstructs with MSE > 0.1 — the model has never seen such patterns and cannot encode them well. + +### Interview Questions + +**Q: How do autoencoders detect anomalies?** +A: Autoencoder-based anomaly detection relies on reconstruction error. The model is trained only on "normal" examples — it learns to compress and reconstruct these patterns efficiently. At inference, when an anomalous input is presented, the encoder compresses it to a latent code that doesn't match any learned pattern, and the decoder reconstructs from this poor encoding, producing a high reconstruction error. The threshold is typically set using the training set's reconstruction error distribution: if training inliers have MSE < 0.01 and outliers have MSE > 0.05, set the threshold at 0.03. This works for high-dimensional data (images, sensor readings, log sequences) where manual feature engineering for anomaly detection is difficult. The method assumes anomalies are rare and qualitatively different from normal data. + +**Q: What is the difference between autoencoders and PCA?** +A: Both learn compressed representations, but: (1) Autoencoders with linear activations and MSE loss learn the same principal subspace as PCA — the encoder weights span the principal components. However, autoencoders can learn non-linear transformations by using non-linear activations (ReLU, tanh), allowing them to capture complex, curved manifolds that PCA cannot represent. (2) PCA components are orthogonal and ordered by variance explained; autoencoder features are not necessarily orthogonal and have no inherent ordering. (3) Autoencoders are more parameter-efficient for complex data — PCA on 224×224 images requires a 50,176×k matrix (inefficient), while a convolutional autoencoder uses shared weights. (4) PCA is deterministic and globally optimal (closed-form SVD); autoencoders use iterative optimization and can find local minima. In practice, autoencoders outperform PCA when the data lies on a non-linear manifold. + +**Q: What causes autoencoders to fail?** +A: Common failure modes: (1) Overcomplete bottleneck — if the bottleneck is larger than the input, the autoencoder can trivially copy input to output without learning useful features, unless regularization is applied (sparsity, denoising, contractive penalty). (2) Overfitting — with too much capacity, the autoencoder memorizes training examples rather than learning generalizable features, resulting in poor latent representations for novel data. (3) Mismatch between training and test distribution — if the test data comes from a different distribution (e.g., autoencoder trained on MNIST digits tested on Fashion-MNIST), reconstruction error is high for all inputs, making anomaly detection impossible. (4) Inadequate bottleneck — if the bottleneck is too small for the data complexity, the autoencoder loses discriminative information. + +--- + +## GAN (Generative Adversarial Network) +**Definition:** A framework where two neural networks — a Generator and a Discriminator — compete in a zero-sum game, with the Generator learning to produce realistic data and the Discriminator learning to distinguish real from fake. + +### Theory & Explanation + +The GAN framework consists of two adversaries: the Generator (G) takes random noise z from a latent distribution (typically N(0,1) in 100-512 dimensions) and produces synthetic data G(z). The Discriminator (D) receives both real data from the training distribution and fake data from the Generator, and must classify each as real or fake. The training objective is a min-max game: min_G max_D V(D,G) = E_x[log D(x)] + E_z[log(1 - D(G(z)))]. The Discriminator tries to maximize this (correctly classify real as real, fake as fake), while the Generator tries to minimize it (make the Discriminator misclassify fakes as real). The equilibrium occurs when the Generator produces data indistinguishable from real data, and the Discriminator is forced to output 0.5 for all inputs (random guessing). + +Training GANs is notoriously unstable for several reasons. The min-max formulation creates a dynamic equilibrium that can oscillate rather than converge. Mode collapse is a common failure where the Generator learns to produce only a few plausible examples (a single digit type or a single face), repeating them to fool the Discriminator. The Discriminator can become too strong, perfectly distinguishing real from fake, providing no gradient signal to the Generator. Conversely, the Generator can exploit any weakness in the Discriminator to consistently win. To address these issues, many architectural and training improvements have been developed. + +Key GAN variants: DCGAN (Deep Convolutional GAN) introduced architectural guidelines (strided convolutions, batch normalization, no fully connected layers) for stable GAN training with CNNs. Conditional GAN (cGAN) conditions both G and D on class labels, enabling controlled generation — generate a specific digit or object category. Wasserstein GAN (WGAN) replaces the standard binary classification loss with Wasserstein distance (Earth Mover's distance) and uses a Critic instead of a Discriminator, eliminating the log function and improving stability. WGAN-GP adds a gradient penalty for the Lipschitz constraint, further improving training. StyleGAN introduces style control through adaptive instance normalization, enabling fine-grained control over generated images (facial features, age, expression). + +### Example + +Training a DCGAN on CelebA (200K celebrity faces, 64×64). Generator input: 100-dimensional noise, output: 64×64 RGB image. After 100 epochs, generated images are recognizable faces but with visible artifacts (blurry eyes, asymmetric features). After 500 epochs, the Generator produces high-quality faces at 128×128 resolution — 96% of generated images are classified as real by humans in a blind test. The latent space shows meaningful interpolation: if you generate a face from latent z₁ (young, smiling) and z₂ (old, neutral), the intermediate latent vectors produce smooth morphing — young→old with gradual wrinkle addition, smile→neutral fading. Arithmetic in latent space works: z_male + z_sunglasses - z_female tends to generate a male with sunglasses, demonstrating that the generator has learned disentangled representations. + +### Interview Questions + +**Q: What is mode collapse and how do you detect it?** +A: Mode collapse is when the Generator produces only a limited variety of outputs, failing to cover the full diversity of the training distribution. For example, a GAN trained on MNIST might only generate digits 0, 1, and 2, ignoring the other seven digits. Detection methods: (1) Visual inspection — generate a grid of 64 random samples; if many look identical or nearly identical, mode collapse is occurring. (2) Inception Score (IS) — measures both quality and diversity of generated images. Low IS for a dataset with high diversity (e.g., ImageNet) suggests mode collapse. (3) Fréchet Inception Distance (FID) — compares feature statistics of real and generated images. FID increases when the Generator misses modes. (4) Training dynamics — the Discriminator loss oscillates or drops to near-zero (Distinguishes real too easily, Generator has no gradient). Solutions: mini-batch discrimination (make Discriminator consider full batch), unrolled GANs, or WGAN loss. + +**Q: Why is the Discriminator-Generator balance critical?** +A: If the Discriminator is too strong (e.g., has much more capacity or faster learning rate), it perfectly distinguishes real from fake, causing the Generator's loss to saturate at 0 — no gradient flows to the Generator, and it stops learning. If the Generator is too strong, the Discriminator collapses to 0.5 for all inputs, and the Generator can "get away" with generating obviously fake data. The optimal training regime requires balancing both networks so that the Discriminator is strong enough to provide useful gradients but weak enough that the Generator can continuously improve. Practical strategies: (1) Train the Discriminator more frequently (e.g., 5 Discriminator updates per Generator update). (2) Use separate learning rates (lower LR for Discriminator). (3) Label smoothing — use 0.9 for real labels instead of 1.0, making the Discriminator less confident. (4) Add noise to Discriminator inputs to prevent it from becoming too sharp. + +**Q: How does WGAN improve training stability?** +A: WGAN makes three critical changes: (1) Replaces the standard log-loss (JS divergence) with Wasserstein distance (Earth Mover's distance), which provides smooth gradients everywhere — even when the Generator produces obviously bad samples, there's a meaningful gradient direction. Standard GAN loss saturates (gradient ≈ 0) when the Discriminator is too confident, but Wasserstein distance always points toward improving the Generator. (2) Removes the sigmoid from the Discriminator's output — the Critic outputs a scalar score (not a probability), representing how "real" the input is. The Generator tries to maximize this score. (3) Enforces Lipschitz continuity (1-Lipschitz constraint) — originally via weight clipping, later improved with gradient penalty (WGAN-GP). This constraint ensures the Critic's output doesn't change too rapidly, providing stable, meaningful gradients. The result: WGAN rarely mode collapses, converges more reliably, and produces higher quality images with the same architecture. + +--- + +## ResNet +**Definition:** Residual Network — a deep CNN architecture that uses skip connections (residual connections) to allow gradients to flow directly through the network, enabling training of very deep networks (50-152+ layers) without vanishing gradients. + +### Theory & Explanation + +The core insight of ResNet is the residual block. Instead of learning a direct mapping H(x), the block learns the residual F(x) = H(x) - x, and the output is H(x) = F(x) + x. The skip connection adds the input x directly to the output of the convolutional layers. Mathematically, for a block with two 3x3 convolutional layers followed by batch normalization and ReLU: output = σ(BN(Conv2(σ(BN(Conv1(x))))) + x). The addition operation + x is the skip connection — it provides a direct path for gradients to flow backward from later layers to earlier layers without being multiplied by weight matrices. If the convolutional layers learn nothing (all weights zero), the block simply passes the input through (F(x) = 0, output = x = identity). + +Skip connections solve the degradation problem, where adding more layers to a plain network causes higher training error (not just test error). This is counterintuitive: deeper networks should be able to approximate any function a shallower network can by making extra layers identity mappings. In practice, plain networks cannot learn identity mappings easily — the weight matrices interact and gradient signals attenuate through many layers. Skip connections provide a shortcut that gives the gradient a "highway" to early layers, bypassing intermediate weight multiplications. During backpropagation, the gradient ∂L/∂x at the bottom of a residual block has a direct additive term ∂L/∂output, which is not multiplied by any weight matrix. + +The ResNet family includes ResNet-18, 34, 50, 101, and 152. For deeper variants (50+), the "bottleneck" block is used: 1×1 convolution (reduce channels) → 3×3 convolution (process) → 1×1 convolution (restore channels). The 1×1 layers reduce computation by compressing the channel dimension from 256 to 64 before the expensive 3×3 convolution, then restoring to 256. A 3-layer bottleneck block has similar computational cost to a 2-layer standard block but is twice as deep, enabling significantly deeper networks. ResNet-152 achieves 152 layers using bottleneck blocks, all trainable with standard SGD due to skip connection gradient flow. ResNet achieved 3.57% top-5 error on ImageNet (2015 ILSVRC winner), surpassing human-level performance for the first time. + +### Example + +Consider ResNet-50 trained on ImageNet (1.28M images, 1000 classes). The network has 50 layers and 25.6M parameters. Total FLOPs: 3.8 billion per image. With standard data augmentation and SGD (LR=0.1, batch=256), it achieves 76.2% top-1 accuracy and 92.9% top-5 accuracy in 90 epochs. The skip connections enable this depth: without them, a 50-layer plain CNN achieves only 68% top-1 accuracy — the degradation problem causes the extra layers to hurt performance. The gradient magnitude at the first convolutional layer of ResNet-50 is ~0.001 × the final layer gradient (healthy flow), while a plain 50-layer network has gradient magnitude ~10^-12 × final layer (effectively no learning in early layers). The bottleneck block reduces computation by 4× compared to a 2-layer 3×3 block at the same channel size. + +### Interview Questions + +**Q: Why do skip connections enable training of much deeper networks?** +A: Skip connections provide a direct gradient highway. In a network without skip connections, the gradient at layer l is: ∂L/∂x_l = ∂L/∂x_L × Π(W_i), the product of weight matrices from layer l to L. With 50 layers, if each weight has spectral norm ~0.9, the gradient product decays as 0.9^50 ≈ 0.005 — early layers receive almost no gradient. With a residual network, the gradient at layer l includes an additive identity term: ∂L/∂x_l = ∂L/∂x_L × (1 + Σ_term). The "1" ensures that even if all paths through weights vanish, the gradient has a direct additive path from the loss to every layer. Think of it as 50 parallel highways, not just one serial road: gradients can bypass any number of intermediate layers. This means early layers receive strong gradient signals throughout training, regardless of total depth. + +**Q: When should you increase network depth vs width?** +A: Depth and width both increase capacity but have different tradeoffs. Increasing depth: (1) Provides more hierarchical feature learning — deeper layers learn more abstract features (edges → textures → parts → objects). (2) Increases effective receptive field — deeper layers aggregate information from larger input regions. (3) Is parameter-efficient — adding depth adds fewer parameters than width for the same capacity increase. However, depth beyond a point provides diminishing returns (ResNet-1001 only marginally outperforms ResNet-152). Increasing width: (1) Provides more parallel feature detectors at each level — better at capturing diverse patterns at the same abstraction level. (2) Higher accuracy for a given computational budget at moderate depths (WideResNet outperforms deep ResNet with same FLOPS). (3) Better parallelization on modern hardware (wide layers use more matrix operations, which GPUs excel at). Guideline: start with standard depth (ResNet-50) and increase width (EfficientNet scales depth, width, and resolution jointly). + +--- + +## U-Net +**Definition:** A symmetric encoder-decoder architecture with skip connections for pixel-level prediction tasks, primarily image segmentation, where the encoder captures context and the decoder recovers spatial details. + +### Theory & Explanation + +U-Net has a distinctive U-shaped architecture with two paths. The encoder (contracting path) follows a typical CNN design: repeated 3×3 convolutions followed by 2×2 max pooling (downsampling), doubling the number of feature channels at each stage. This reduces spatial dimensions (224×224 → 112×112 → 56×56 → 28×28 → 14×14) while increasing feature depth (32 → 64 → 128 → 256 → 512 channels). The encoder captures context — it answers "what is in the image" by building increasingly abstract feature representations. Each downsampling step loses spatial resolution but gains semantic understanding. + +The decoder (expanding path) mirrors the encoder: it upsamples feature maps using transposed convolutions (2×2, stride 2) to progressively restore spatial resolution (14×14 → 28×28 → 56×56 → 112×112 → 224×224). At each upsampling stage, the decoder concatenates its upsampled features with the corresponding encoder features via skip connections. These skip connections are the key innovation — they pass high-resolution spatial information from the encoder directly to the decoder, bypassing the bottleneck where spatial details are lost. Without skip connections, the decoder would have only the compressed (14×14) information, producing coarse segmentations with lost boundaries. + +The final layer is a 1×1 convolution with softmax (or sigmoid for binary) to produce pixel-wise class probabilities. For a binary segmentation task (foreground/background), the output has 2 channels. For multi-class, it has C channels (C = number of classes). The loss is typically pixel-wise cross-entropy or Dice loss (for imbalanced classes, common in medical imaging where foreground is small). U-Net requires relatively few training images (as few as 30-50 pairs) to produce good segmentations due to its efficient use of data through the skip connections and data augmentation. + +### Example + +Applying U-Net to medical image segmentation: segmenting lung nodules in CT scans (512×512, binary: nodule vs background). The encoder downsamples through 4 stages (512→256→128→64→32) with 32→64→128→256→512 channels. The decoder upsamples back to 512×512. Each skip connection concatenates encoder features (e.g., 256×256 with 64 channels) with upsampled decoder features (256×256 with 128 channels), giving the decoder 192 channels with both high-level context and fine-grained spatial detail. Training on 50 CT scans (augmented to 500 with rotations, elastic deformations) for 100 epochs achieves 0.94 Dice score. Without skip connections, the same architecture achieves only 0.78 Dice — the model loses small nodules (size < 5mm) because spatial resolution in the bottleneck is insufficient to preserve small details. + +### Interview Questions + +**Q: Why are skip connections particularly important for segmentation?** +A: Segmentation requires both understanding "what" an object is (semantic context) and "where" its boundaries are (spatial precision). The encoder excels at "what" by building abstract, high-channel feature representations through downsampling. However, downsampling destroys spatial information — at 14×14 resolution, a 28×28 feature region corresponds to one pixel, making boundary localization impossible at this scale. Skip connections give the decoder direct access to high-resolution encoder features (112×112, 56×56) that preserve pixel-level spatial information. The decoder can combine deep semantic features ("this is a lung") with shallow spatial features ("this edge gradient marks the lung boundary"). This dual access to both types of information is what makes U-Net exceptionally good at segmentation. + +**Q: How does U-Net handle different-sized objects?** +A: U-Net's multi-resolution feature maps naturally handle multi-scale objects. Small objects (e.g., nodules < 10 pixels) are best captured in the early encoder features (high resolution) and passed via early skip connections — the decoder at the corresponding resolution level has fine-grained spatial information. Large objects (e.g., an entire organ) are captured in later encoder features (low resolution, high channel depth) where the aggregated receptive field covers the full object. The decoder at each level combines the appropriate resolution for both small and large structures. For very small objects or thin structures (e.g., blood vessels, cell membranes), researchers add deeper skip connections or use attention gates that weight which spatial features to emphasize at each decoder level. + +--- + +## Transfer Learning +**Definition:** A technique where a model trained on one task (typically large-scale, general domain) is reused as the starting point for a related task, leveraging learned features to improve performance with limited data and reduce training time. + +### Theory & Explanation + +Transfer learning exploits the hierarchical nature of deep neural networks. Early layers learn general features that are useful across many tasks (edges, textures, color blobs for vision; syntax, morphology for NLP). Later layers learn task-specific patterns. By starting from pre-trained weights rather than random initialization, the model has already learned these universal feature extractors. Only the task-specific layers need significant adaptation, dramatically reducing the amount of data and compute needed for the new task. The pre-trained model provides a strong prior that prevents overfitting when the target dataset is small. + +Two main approaches: Feature Extraction (FE) and Fine-Tuning (FT). Feature Extraction freezes the pre-trained backbone (no gradient updates) and only trains newly added classifier layers on top. This treats the pre-trained model as a fixed feature encoder — the new task learns to interpret the extracted features. FE is faster (fewer trainable parameters, no backward pass through the backbone) and uses less memory. Fine-Tuning trains all or some of the pre-trained layers with a small learning rate (typically 1e-5 to 5e-5, 10-100× lower than the initial pre-training rate). FT allows the pre-trained features to adapt to the new task, which works better when the target domain differs significantly from the pre-training data. + +When to use each approach depends on dataset size and similarity. If the target dataset is small (< 1000 examples) and similar to the pre-training data: use Feature Extraction or fine-tune only the last 1-2 layers. If the target dataset is large (> 10,000 examples) and different from pre-training data: fine-tune the entire model. If the target dataset is large and similar: fine-tune all layers with a low learning rate. If the target dataset is small and different: consider training from scratch or using a different pre-trained model that is closer to the target domain. The learning rate for fine-tuning is typically 1/10 to 1/100 of the original pre-training rate to avoid catastrophic forgetting. + +### Example + +Fine-tuning ResNet-50 (pre-trained on ImageNet, 1000 classes) for a medical imaging task — classifying chest X-rays (pneumonia vs normal, 5,860 images). Option 1 (Feature Extraction): freeze the convolutional backbone, add a new classification head (average pooling → FC 2048 → FC 128 → FC 2). Train only the head for 20 epochs at LR=0.01. Achieves 87% accuracy — reasonable because ImageNet features (edges, textures) transfer well, but the specific X-ray patterns are not fully captured. Option 2 (Fine-Tuning): initialize with ImageNet weights, replace the final classification layer (1000 → 2), fine-tune all layers at LR=1e-4 for 20 epochs. Achieves 92% accuracy — the backbone adapts from natural image features to X-ray features. Option 3 (Scratch): train from random initialization at LR=0.1 for 100 epochs. Achieves only 75% accuracy and requires 5× more compute. Transfer learning saves 80% of training time and improves accuracy by 17%. + +### Interview Questions + +**Q: What is catastrophic forgetting and how do you prevent it?** +A: Catastrophic forgetting occurs when fine-tuning causes the model to overwrite the pre-trained features learned from the original task. If the new task learning rate is too high, the backbone weights move far from their pre-trained values, losing the general knowledge that made transfer learning valuable. For example, if you fine-tune BERT on a small sentiment analysis dataset at LR=2e-4 (50× the recommended rate), the model will quickly fit the training set but lose its general language understanding — perplexity on general text drops significantly. Prevention strategies: (1) Use low learning rates (1e-5 to 5e-5 for BERT fine-tuning). (2) Freeze early layers and only fine-tune the last 1-3 layers. (3) Use gradual unfreezing — train the head first, then gradually unfreeze lower layers. (4) Add regularization (ELASTIC weight consolidation, knowledge distillation from the pre-trained model). (5) Use a smaller learning rate for backbone than head (differential learning rates). + +**Q: When should you NOT use transfer learning?** +A: Transfer learning can be ineffective or harmful when: (1) The target task is fundamentally different from the pre-training task. Example: a model pre-trained on natural images (ImageNet) for medical CT scans — the features (fur, wheels, water) don't correspond well to medical patterns (tissue textures, anatomical structures). Medical models often need CT-specific pre-training. (2) The target dataset is very large and you have sufficient compute — training from scratch can match or exceed transfer learning performance by customizing the architecture and training procedure to the specific task. (3) The pre-trained model contains biases that hurt the target task — for instance, pre-trained word embeddings often encode gender or racial biases that can propagate to downstream tasks. (4) Domain mismatch is too severe — pre-trained on English text, fine-tuned on code or mathematical notation. (5) You need strict data/policy compliance — pre-trained models may have been trained on data that cannot be used for your application. + +--- + +# 5. NLP (Advanced) + +## Sentence Embeddings +**Definition:** Dense vector representations of entire sentences that capture semantic meaning in a fixed-length numerical format, enabling comparison, clustering, and retrieval of sentences by meaning. + +### Theory & Explanation + +Sentence embeddings are the sentence-level analog of word embeddings — mapping variable-length sentences to fixed-dimensional vectors (typically 384-768 dimensions) where semantically similar sentences have similar vectors. Unlike taking the average of word embeddings (which loses word order and context), sentence embedding models are trained specifically to produce meaningful sentence-level representations. The key requirement is that the embedding space reflects semantic similarity: "The dog chased the cat" should be close to "The cat was chased by the dog" but far from "I like pizza." + +Sentence-BERT (SBERT) is the most widely used approach. It fine-tunes BERT/RoBERTa using siamese and triplet networks on Natural Language Inference (NLI) data. In the siamese architecture, two identical BERT networks (shared weights) encode two sentences independently, producing two embeddings u and v. Training uses the SNLI and MultiNLI datasets (570K+ premise-hypothesis pairs labeled as entailment, contradiction, or neutral). The loss functions include: (1) Classification objective: concatenate (u, v, |u-v|) and classify the relationship. (2) Cosine similarity loss: maximize cosine similarity for entailment pairs, minimize for contradiction pairs. (3) Triplet loss: anchor positive (similar sentence) closer than anchor negative (dissimilar sentence) by a margin. + +The resulting embeddings capture sentence-level semantics remarkably well. SBERT variants (all-MiniLM-L6-v2, all-mpnet-base-v2) produce 384-768 dimensional embeddings that enable nearest-neighbor search over millions of sentences in milliseconds using FAISS or other approximate nearest neighbor libraries. Universal Sentence Encoder (USE) from Google takes a different approach, using a Transformer trained with multi-task objectives (skip-thought, conversational response prediction, natural language classification). USE produces 512-D embeddings and is faster for inference but typically less accurate than SBERT for semantic similarity tasks. + +### Example + +Building a semantic search engine for customer support queries. Collect 50,000 support tickets. Encode each ticket using SBERT (all-MiniLM-L6-v2, 384-dimensional) — takes ~5 minutes on GPU. Store vectors in FAISS index. A new query "How do I reset my password?" is encoded to the same embedding space. FAISS finds the top-5 most similar tickets in <10ms. Cosine similarity scores: the most relevant ticket "Steps to reset account password" scores 0.92, a somewhat related ticket "Forgot my login credentials" scores 0.71, and an unrelated ticket "How to upgrade plan" scores 0.12. Without SBERT, using average of GloVe word embeddings, the same query returns "Password security best practices" (0.65 score) as the top result — semantically wrong because word averaging loses the "how to reset" intent. + +### Interview Questions + +**Q: Why is taking the average of BERT's word embeddings a poor sentence embedding?** +A: BERT was not trained to produce meaningful sentence-level representations — it was trained for token-level masked language modeling and next sentence prediction. The average of all token embeddings from BERT suffers from several problems: (1) Each token's embedding is heavily influenced by its local context, so the average is dominated by the most frequent or extreme tokens, not the overall meaning. (2) BERT's token representations are not isotropic — they occupy a narrow cone in the embedding space, making cosine similarity unreliable (all sentence averages end up with high cosine similarity to each other regardless of actual similarity). (3) The [CLS] token (often used as sentence representation) was found to encode primarily the sentence's topic but not fine-grained semantic relationships — SBERT showed that [CLS] averages from fine-tuned BERT correlate only 0.2-0.3 with human similarity judgments. SBERT specifically fine-tunes to create an isotropic, semantically meaningful embedding space, achieving 0.75-0.85 correlation with human judgments. + +**Q: How do you handle OOV (out-of-vocabulary) tokens in sentence embeddings?** +A: Modern sentence embedding models use subword tokenization (WordPiece, BPE, SentencePiece), which essentially eliminates true OOV tokens. Any unseen word is broken into known subword units: "embiggen" → "em" + "##big" + "##gen". Sentence embedding models encode at this subword level and aggregate to produce a sentence embedding. However, rare word embeddings may be poorly estimated. To mitigate: (1) Use a model trained on diverse data (covers more subword combinations). (2) For domain-specific terms, fine-tune the sentence embedding model on in-domain data. (3) For production systems, use a model with a larger vocabulary. The tradeoff: larger vocabulary = more parameters, slower inference. In practice, modern models handle OOV well and subword decomposition rarely causes significant performance degradation. + +--- + +## NER (Named Entity Recognition) +**Definition:** The task of identifying and classifying named entities in text into predefined categories such as Person, Organization, Location, Date, Time, Money, and Percentage. + +### Theory & Explanation + +NER is typically formulated as a sequence labeling task: given a sequence of tokens, assign each token a label representing the entity type and position (BIO/IOB2 tagging scheme). B-PER marks the Beginning of a Person entity, I-PER marks Inside (continuation of) a Person entity, O marks Outside (not an entity). Example: "[John]B-PER [Smith]I-PER works at [Google]B-ORG in [Mountain View]B-LOC [California]I-LOC." This scheme allows the model to recognize multi-token entities and distinguish between adjacent entities of the same type. + +Traditional approaches use BiLSTM-CRF (Bidirectional LSTM + Conditional Random Field). The BiLSTM encodes each token with both left and right context, producing a context-rich hidden representation for each position. The CRF layer on top models dependencies between adjacent labels — it learns that I-PER should not follow B-LOC without an intervening O, and B-PER should be followed by I-PER or O (not B-ORG without O). The CRF's transition matrix enforces valid label sequences, improving accuracy by 1-3% over independent token classification (softmax per position). The Viterbi algorithm decodes the optimal label sequence exactly. + +Modern approaches use Transformer-based models (BERT, RoBERTa, etc.) fine-tuned for token classification. The model takes a sentence, produces contextualized token embeddings, and a linear classifier predicts the label for each token. BERT-based NER achieves state-of-the-art results (F1 > 92% on CoNLL-2003) due to its deep bidirectional context. The main challenge is entity boundary detection — identifying exactly where an entity starts and ends. Evaluation uses exact match (predicted span must exactly match gold span) and partial match (overlap counts). Agreement between human annotators on NER is typically around 95-97%, setting an upper bound on achievable accuracy due to genuine ambiguity. + +### Example + +Processing a news article: "Apple Inc. CEO Tim Cook announced the new iPhone 15 at Apple Park in Cupertino on September 12, 2023, for $999." A BERT-based NER model tokenizes and labels: [Apple Inc.]ORG, [Tim Cook]PER, [iPhone 15]PRODUCT, [Apple Park]LOC, [Cupertino]LOC, [September 12, 2023]DATE, [$999]MONEY. The confidence scores: "Apple Inc." as ORG: 0.97, "Tim Cook" as PER: 0.99, "Apple Park" as LOC: 0.88 (note: "Apple" could be ORG, but context confirms it's a location). The model correctly disambiguates "Apple" as ORG in "Apple Inc." and LOC in "Apple Park." A BiLSTM-CRF model scores 88% F1 on this type of text, while fine-tuned BERT scores 93% F1, with the main improvement in handling ambiguous entities (like "Apple") and long, rare entities. + +### Interview Questions + +**Q: Why does the CRF layer improve NER over token-level softmax classification?** +A: CRF captures label dependencies that a token-level classifier (softmax per token) ignores. Consider the sequence: "New York City." A token-level classifier might predict: B-LOC (New), B-LOC (York), B-LOC (City) — each token independently chooses B-LOC with high probability. But this violates the BIO scheme: York should be I-LOC after B-LOC, and City should be I-LOC. The CRF learns transition probabilities between labels: P(B-LOC → I-LOC) is high (0.95), but P(B-LOC → B-LOC) is low (0.01) because two entities of the same type should not be adjacent without O. When the token-level model assigns B-LOC to both "New" and "York" with 0.7 probability, the CRF reweights: the sequence (B, I, I) gets CRF score (0.7 × 0.95 × 0.95), while (B, B, B) gets score (0.7 × 0.01 × 0.01) — 100× lower. This constraint-based reasoning improves F1 by 1-3%. + +**Q: How do you handle nested entities?** +A: Nested entities (e.g., "[University of [California]ORG]ORG") are challenging for standard BIO tagging, which cannot represent hierarchies. Approaches: (1) Layered NER — run multiple passes: first extract top-level entities ("University of California"), then scan within them for sub-entities ("California"). (2) Hypergraph representations — explicitly model entity hierarchies in the label space. (3) Span-based methods — instead of token-level tagging, enumerate all possible spans and classify each span as an entity type or none. Span-based methods handle nesting naturally because a span can be inside another span. For example, "University of California" is span [1-3] classified as ORG, and "California" is span [3-3] also classified as ORG. (4) Sequence-to-sequence models — generate entity labels as a structured output, where nesting is represented in the output format. In practice, BERT-based span classification achieves the best performance for nested NER. + +**Q: How do you handle overlapping and ambiguous entities?** +A: Ambiguity resolution depends on context: "Washington" could be PER (George Washington), LOC (Washington state), ORG (Washington Post), or GPE (Washington, D.C.). Context disambiguation: if preceded by "President" → PER; if followed by "D.C." → GPE; if followed by "announced that" → could be GPE or ORG. Modern BERT models handle most ambiguity naturally (90%+ accuracy) through contextual embeddings. For remaining challenging cases: (1) Use domain-specific knowledge bases or gazetteers to bias predictions. (2) Add entity linking — link mentioned entity to a knowledge base (Wikipedia) to resolve identity. (3) Use coreference resolution — "He traveled to Paris. The city has great food" confirms Paris is a LOC, not PER. (4) For genuinely ambiguous cases with insufficient context, the model should output low confidence or multiple possible entities, letting downstream systems handle the uncertainty. + +--- + +## POS Tagging +**Definition:** Part-of-Speech tagging — the task of assigning grammatical tags (noun, verb, adjective, etc.) to each word in a text based on its definition and context. + +### Theory & Explanation + +POS tagging is a foundational NLP task that serves as input to many higher-level systems. Each word receives a single tag from a predefined tagset. The Universal Dependencies tagset has 17 coarse tags (NOUN, VERB, ADJ, ADV, PRON, DET, ADP, etc.) that work across languages. More detailed tagsets like the Penn Treebank (45-50 tags) distinguish fine-grained categories: NNP (proper noun, singular), NNS (noun, plural), VBD (verb, past tense), VBG (verb, gerund/present participle). The ambiguity is substantial: "run" can be VERB ("I run fast"), NOUN ("a morning run"), or ADJ ("run time"). + +Traditional statistical approaches include Hidden Markov Models (HMMs), which model the sequence of tags as a Markov chain. HMMs use transition probabilities P(tag_i | tag_{i-1}) and emission probabilities P(word_i | tag_i). The Viterbi algorithm finds the most likely tag sequence given the observed words. HMMs achieve ~90% accuracy on English. Maximum Entropy Markov Models (MEMMs) improve on HMMs by allowing arbitrary features (word shape, capitalization, suffix) but suffer from label bias (prefer fewer tag transitions). Conditional Random Fields (CRFs) solve the label bias problem by globally normalizing the tag sequence probability: P(Y|X) = exp(Σfeatures) / Z(X), where Z is the partition function. + +Modern approaches use BiLSTM-CRF or Transformer-based models. A BiLSTM reads the sentence in both directions, producing a context-rich representation for each position. The CRF layer on top models tag-transition constraints. Fine-tuned BERT achieves >97% accuracy on English POS tagging (WSJ/Penn Treebank), approaching human inter-annotator agreement (~98%). The remaining errors are genuinely ambiguous cases: "I saw her duck" — duck as NOUN vs VERB requires real-world knowledge. For cross-lingual POS tagging, multilingual BERT (mBERT) and XLM-R achieve strong zero-shot transfer, with accuracies >90% on languages seen during training and 80-85% on unseen languages. + +### Example + +Tagging the sentence: "The quick brown fox jumps over the lazy dog." A BERT-based tagger produces: The/DET, quick/ADJ, brown/ADJ, fox/NOUN, jumps/VERB, over/ADP, the/DET, lazy/ADJ, dog/NOUN. Confidence scores: all tags >0.98 — straightforward case. Now consider: "I saw her duck under the table." Our model produces: I/PRON, saw/VERB, her/PRON, duck/VERB, under/ADP, the/DET, table/NOUN — confidence for "duck" as VERB is 0.72, NOUN is 0.28. The model uses the context "under the table" (preposition indicates location) and the absence of a determiner before "duck" (no article, so likely verb, not noun). With HMM, the same sentence gets duck tagged as NOUN (probability 0.55 for NOUN vs 0.45 for VERB) — the HMM can't leverage the full context to make the correct decision. + +### Interview Questions + +**Q: Why is POS tagging more difficult than it seems?** +A: POS tagging is deceptively challenging because: (1) Ambiguity is pervasive — over 40% of English word types have multiple possible POS tags. "Bank" is NOUN (financial institution), VERB (to bank a check), or NOUN/VERB (river bank). "Well" can be NOUN, ADV ("well done"), ADJ ("I'm well"), or INTJ ("Well!"). (2) Unknown words — words not seen during training require morphological analysis: "-ly" often indicates ADVERB, "-tion" indicates NOUN, "-ed" suggests VERB past tense. For languages with rich morphology (Finnish, Turkish), this is much harder. (3) Cross-lingual variation — languages have fundamentally different syntactic structures. Japanese has postpositions instead of prepositions; Chinese lacks overt tense marking. Tagsets need to work across these differences while remaining linguistically meaningful. (4) Domain adaptation — financial text and social media have different linguistic patterns. "Like" in "He was like, wow" (discourse marker) vs "I like this" (VERB). State-of-the-art models now handle most of these challenges with contextual embeddings. + +**Q: How do you evaluate a POS tagger?** +A: Standard evaluation metrics: (1) Per-token accuracy — percentage of words tagged correctly. Most straightforward and common metric. Typical scores: HMM ~95%, BiLSTM-CRF ~96.5%, BERT ~97.5% on Penn Treebank. (2) Per-sentence accuracy — percentage of sentences where every word is tagged correctly. Much harder: 50-55% for best systems — a single error in a 20-word sentence fails the whole sentence. (3) Unknown word accuracy — accuracy only on words not seen during training, which is typically 5-15% lower than overall accuracy. This separates methods that rely on memorization from those that use morphological and contextual cues. (4) Category-specific accuracy — some categories are naturally harder: adverbs and particles typically have lower accuracy than nouns and verbs. Evaluation datasets: Penn Treebank WSJ sections for English, Universal Dependencies treebanks for cross-lingual evaluation. + +--- + +## Sentiment Analysis +**Definition:** The task of determining the emotional tone or opinion expressed in text — typically classifying polarity (positive, negative, neutral) or predicting a numerical sentiment score. + +### Theory & Explanation + +Sentiment analysis operates at multiple granularities: document-level (entire review sentiment), sentence-level (each sentence's polarity), aspect-level (specific aspects of a product: "battery life" positive, "screen" negative), and fine-grained (1-5 star ratings, or continuous -1 to +1 score). The choice of granularity depends on the application: a movie review might be generally positive (4 stars) but negative about "the pacing" (aspect-level). Aspect-based sentiment analysis is the most informative but most challenging, requiring both entity extraction and sentiment classification per entity. + +Approaches span three generations. Lexicon-based (VADER, TextBlob): use pre-built sentiment dictionaries with valence scores (+1.0 for "excellent", -0.5 for "disappointing"). Apply rules: "not good" negates positive, intensifiers ("very good") amplify score. VADER handles social media (emoticons, slang, capitalization) well. Accuracy: ~65-70% on general text — good for zero-shot, limited for nuanced language. Machine learning approach: extract features (TF-IDF, n-grams, word embeddings) and train a classifier (Logistic Regression, SVM, Random Forest). This requires labeled data but achieves 80-85% accuracy. Logistic Regression with TF-IDF on bigrams is remarkably effective for document-level sentiment. + +Deep learning approach: fine-tune BERT/RoBERTa for sentiment classification. BERT achieves 90-95% accuracy on standard benchmarks (SST-2, IMDb). The model takes the full text, and the [CLS] token representation feeds a linear classifier. Contextual understanding captures subtle sentiment: "The movie was so bad it was good" — BERT correctly classifies as positive (enjoyment of a "so bad it's good" experience), while bag-of-words models see "bad" and "good" equally weighted and classify neutral or incorrectly. Aspect-based sentiment requires an additional step: encode the aspect term and the sentence together (e.g., BERT input: "[CLS] battery life [SEP] The phone has great battery life but awful camera [SEP]") and classify the sentiment of the aspect. + +### Example + +Building a product review sentiment analyzer. Model: fine-tuned DistilBERT on 50K Amazon reviews (5 categories). Test sentence: "The camera takes amazing photos but the battery barely lasts three hours." Document-level classification: DistilBERT predicts negative (0.72 confidence), correctly focusing on the negative aspect despite positive language about the camera. Aspect-level: the model separately predicts camera→positive (0.93) and battery→negative (0.89) — much more useful for the manufacturer than a single sentiment score. VADER's lexicon approach classifies "amazing" (+1.0) and "barely" (-0.5) combined → compound score of 0.38 (positive) — it fails because it can't distinguish aspects. Naive Bayes with TF-IDF unigrams classifies negative (0.55), also missing the mixed aspects but getting the overall tone correct due to more negative terms. + +### Interview Questions + +**Q: How do you handle sarcasm and figurative language in sentiment analysis?** +A: Sarcasm is one of the hardest challenges for sentiment analysis. "Great, another meeting that could have been an email" — the words are positive ("great") but the intent is negative. Approaches: (1) Contextual models — BERT-based classifiers leverage the full context and are surprisingly effective (70-75% sarcasm detection accuracy), as they learn that "great" followed by complaint patterns signals sarcasm. (2) Sarcasm-specific fine-tuning — train on sarcasm-annotated datasets (SARC, iSarcasmEval). (3) Multi-modal cues — for social media posts, incorporate emojis \[expression\], punctuation (!!!), and capitalization (GREAT), which are strong indicators of sarcasm. (4) Contrastive approaches — if there's a sentiment mismatch between literal word meaning and context, flag as potential sarcasm. (5) Ensemble models — combine sentiment classifier and sarcasm detector: if high-confidence sarcasm, invert or adjust sentiment score. Despite these methods, sarcasm remains challenging — human agreement on sarcasm annotation is only ~85%, setting an upper bound. + +**Q: What is aspect-based sentiment analysis and why is it harder?** +A: Aspect-based sentiment analysis (ABSA) identifies aspects (product features, service attributes) mentioned in text and assigns a sentiment to each. Steps: (1) Aspect extraction — identify the aspect terms: "battery life," "camera," "price" from reviews. (2) Aspect sentiment classification — classify each aspect's sentiment. Harder than document-level because: (a) Multiple aspects can appear in one sentence with conflicting sentiments. "The food was delicious but the service was terrible" — food positive, service negative. Document-level analysis would average to neutral, losing both insights. (b) Implicit aspects — "The phone died in two hours" implies battery life without stating it explicitly. (c) Compound aspects — "price-to-performance ratio" is a single aspect but contains multiple sub-aspects. (d) Aspect-dependent sentiment — "The screen is bright" is positive for a phone but neutral/negative for a movie theater. Modern approaches use BERT with separate encoding of the aspect and sentence, achieving 80-85% F1 on ABSA benchmarks. + +--- + +## Text Classification +**Definition:** The task of assigning predefined categories or labels to text documents, including spam detection, topic labeling, intent classification, and content moderation. + +### Theory & Explanation + +Text classification is one of the most widely deployed NLP applications. Formally, given a document d and a set of categories C, the model predicts the most likely category c ∈ C. Binary classification (spam/not-spam) and multi-class (news topic: sports/politics/tech/entertainment) are common. Multi-label classification (assign multiple labels: a document could be both "technology" and "business") requires sigmoid output with binary cross-entropy loss per label. The fundamental challenge is mapping variable-length, unstructured text to a fixed set of categories while handling vocabulary mismatch, domain-specific language, and class imbalance. + +Traditional text classification uses a bag-of-words representation with TF-IDF weighting and a linear classifier. The document is converted to a sparse vector where each dimension corresponds to a word (or n-gram), and the value is the TF-IDF weight: TF(t,d) × IDF(t). A Linear SVM or Logistic Regression classifier learns a weight for each word per class. This approach is surprisingly effective for well-defined categories: spam detection achieves 99%+ accuracy with TF-IDF + SVM. Pros: interpretable (top weighted words per class reveal model reasoning), fast training, low memory. Cons: cannot handle word order, synonyms ("car" vs "automobile" treated as separate features), or nuanced language. + +Deep learning approaches (fastText, CNNs, BERT) improve on bag-of-words. fastText extends word2vec to classification by averaging word embeddings and using hierarchical softmax — extremely fast (train on 1B tokens in 10 minutes) and handles rare words well. CNN-based text classification (Kim CNN) uses 1D convolutions over word embeddings with multiple filter sizes to capture n-gram patterns. BERT fine-tuning achieves state-of-the-art (90-97% on most benchmarks) by using deep bidirectional context. For long documents (>512 tokens), strategies include: truncating to the first 512 tokens, hierarchical attention (split into segments, encode independently, then aggregate), Longformer (extends attention to 4096 tokens), or sliding window + pooling. + +### Example + +Building a spam classifier for email. Dataset: 50K emails (5K spam, 45K ham — imbalanced 10:1). Method 1: TF-IDF (unigrams + bigrams) + Logistic Regression. Top features for spam: "free" (weight 2.1), "click here" (1.8), "limited offer" (1.7), "congratulations" (1.5). Top features for ham: "meeting" (1.2), "attached" (1.1), "regards" (1.0). F1 score: 0.97 for ham, 0.92 for spam. Method 2: fine-tuned DistilBERT. Input truncated to 128 tokens per email. F1 score: 0.99 for ham, 0.97 for spam. BERT catches subtle spam: "You won a prize!!! Click here!!!" — high spam confidence (0.99), but also catches "You've been selected, see attached for details" (0.85) even though individual words aren't obvious spam signals. However, BERT training takes 2 hours on GPU vs 5 minutes for TF-IDF. The practical choice: TF-IDF for quick deployment with good accuracy, BERT when the cost of misclassification is high. + +### Interview Questions + +**Q: How do you handle class imbalance in text classification?** +A: Class imbalance (spam 10%, sentiment negative only 5%) is common in text classification. Mitigation strategies: (1) Resampling — oversample minority class (duplicate examples) or undersample majority class (random removal). Oversampling can lead to overfitting; undersampling loses data. SMOTE generates synthetic samples in feature space (works for TF-IDF, less natural for BERT). (2) Weighted loss — assign higher weight to minority class examples: weight = total_samples / (n_classes × class_count). For 90:10 split, minority class weight = 5.0, majority = 0.56. This makes the model more sensitive to minority class errors. (3) Threshold tuning — find optimal decision threshold using precision-recall curves instead of default 0.5. For imbalanced sets, the optimal threshold for the minority class may be 0.2-0.3. (4) Focal loss — modifies cross-entropy to focus on hard-to-classify examples: FL(p_t) = -(1-p_t)^γ × log(p_t), where γ=2 reduces the loss contribution of well-classified examples. (5) Data augmentation — generate additional minority class examples using synonym replacement or back-translation. + +**Q: What is zero-shot text classification and when would you use it?** +A: Zero-shot classification uses a pre-trained model to classify text into categories it was never explicitly trained on, typically by reformulating classification as natural language inference (NLI) or textual entailment. Example: given premise "I loved this movie, great acting" and hypothesis "This review is positive," the model predicts whether the premise entails the hypothesis. Zero-shot NLI-based classifiers (using BART or DeBERTa fine-tuned on MNLI) achieve 70-85% accuracy on unseen categories. Use cases: (1) Rapid prototyping — when you need to test a classification idea without collecting labeled data. (2) Dynamic categories — topics that change frequently (news categorization, emerging product categories) where retraining is impractical. (3) Low-resource settings — no labeled data exists. The main limitation: zero-shot accuracy is 10-20% lower than fine-tuned models, and the model struggles with categories that require specific domain knowledge. + +--- + +## Machine Translation +**Definition:** The task of automatically translating text from one language (source) to another (target) while preserving meaning, style, and fluency. + +### Theory & Explanation + +Machine translation has evolved through three major paradigms. Statistical Machine Translation (SMT, pre-2014) used phrase-based models: the translation is factored into P(target|source) ∝ P(source|target) × P(target), where the translation model P(source|target) is learned from parallel corpora (aligned sentence pairs) and the language model P(target) ensures fluency. SMT required massive feature engineering: phrase tables with millions of entries, reordering models for word order differences, lexical weighting. Building a production SMT system for a language pair required months of engineering effort. + +Neural Machine Translation (NMT) using sequence-to-sequence (seq2seq) with attention (2014-2017) revolutionized the field. The architecture: an encoder RNN processes the source sentence into a sequence of hidden states, then a decoder RNN generates the target translation one word at a time, attending to relevant source positions at each step. The attention mechanism computes a weighted average of encoder states: context_t = Σα_{t,s}h_s, where α_{t,s} = softmax(score(h_t_dec, h_s_enc)). This allowed the decoder to flexibly align target words to source words without explicit alignment tables. NMT improved translation quality dramatically, especially for fluency (human judgments +50% BLEU over SMT). + +The Transformer (Vaswani et al., 2017) replaced RNNs entirely with self-attention and position-wise feedforward layers, enabling parallel processing of all positions and training on much larger datasets. Google Neural Machine Translation (GNMT) combined the Transformer with engineering: 8 encoder/decoder layers, 1024 hidden size, 16 attention heads, wordpiece tokenization, trained on 100M+ sentence pairs. GNMT reduced translation errors by 60% compared to SMT. Modern systems use pretrained encoder-decoder models like mT5, NLLB (No Language Left Behind — 200 languages), and M2M-100. BLEU score (Bilingual Evaluation Understudy) is the standard metric: BLEU = BP × exp(Σ(1/4 × log(precision_n))), where precision_n measures n-gram overlap (n=1..4) between candidate and reference translations. + +### Example + +Translating English to German: "The cat sat on the mat." SMT: "Die Katze saß auf der Matte." — correct, handled by phrase table. More complex: "The man who is wearing a red hat is my uncle." SMT: "Der Mann, der einen roten Hut trägt, ist mein Onkel." — correct word order is challenging for SMT because English has right-branching relative clauses but German has different word order in subordinate clauses. SMT might produce "Der Mann, der trägt einen roten Hut, ist mein Onkel" (missing the verb-final rule in German subordinate clauses) — BLEU 0.75. NMT/Transformer: "Der Mann, der einen roten Hut trägt, ist mein Onkel." — grammatically perfect, BLEU 1.0. For a rare word not seen in training ("magnetoencephalography"), SMT copies it unknown, while NMT/Transformer produces "Magnetoenzephalographie" — correctly translating the composite. Modern systems achieve BLEU scores of 30-45 for high-resource pairs (En→De, En→Fr) but only 5-15 for low-resource languages (Hausa, Quechua). + +### Interview Questions + +**Q: How does attention help machine translation decode the correct alignment?** +A: Attention computes a weight for each source position at each decoding step, creating a soft alignment. For translating "The cat sat on the mat" to "Die Katze saß auf der Matte," the attention weights for the word "sat" in the decoder would be distributed as: "sat" → 0.80 for "sat" (exact alignment), "cat" → 0.05, "on" → 0.05, etc. For "matte" at the end of the German sentence, attention would focus primarily on "mat" (0.85) with secondary weight on "on" (0.10) because "auf der Matte" corresponds to "on the mat." The attention mechanism doesn't just find the corresponding word — it provides context from the entire sentence weighted by relevance. For long-distance reorderings: English "The man who...is my uncle" → German "Der Mann, der...ist mein Onkel," the decoder generating "ist" (is) must attend back to "uncle" early in the English sentence — attention enables this long-distance mapping that was extremely difficult for SMT. + +**Q: What are the main challenges for low-resource language translation?** +A: Low-resource translation (pairs with <1M parallel sentences, or <100K) faces: (1) Data scarcity — parallel datasets for under-resourced languages are often only thousands or tens of thousands of sentences, insufficient for training large transformer models. (2) Domain mismatch — the limited data is often from narrow domains (religious texts, government documents) that don't represent everyday language. (3) Morphological complexity — many low-resource languages are morphologically rich (Finnish has 15+ noun cases, Navajo is polysynthetic). The limited data doesn't cover the full morphological paradigm. (4) Script variation — many languages use non-Latin scripts, and models need to learn character/script mappings from limited data. Solutions: multilingual models (mT5, NLLB) trained on many languages benefit from cross-lingual transfer; unsupervised machine translation (back-translation using monolingual data); data augmentation via word replacement and sentence mixing; joint BPE training across related languages to share subword units. + +**Q: How would you evaluate machine translation beyond BLEU?** +A: BLEU has known limitations: prefers short translations, correlates imperfectly with human judgment, requires multiple references for reliability, and doesn't capture semantic adequacy. Better metrics: (1) BLEU — recommended as a primary metric only for tracking relative improvements. (2) chrF — character n-gram F-score, less sensitive to tokenization differences, correlates better with human judgment (especially for morphologically rich languages). (3) COMET — neural learned metric using XLM-R to compare source, translation, and reference; achieves 0.85+ correlation with human judgment (vs BLEU's 0.35-0.50). (4) Human evaluation — best but expensive. Metrics: adequacy (does it preserve meaning?) on 1-5 scale, fluency (is it natural?) on 1-5 scale. (5) Task-specific evaluation — for translation used in downstream tasks (e.g., QA), evaluate end-task performance. Practical recommendation: use BLEU for rapid iteration and development, COMET for final evaluation, human spot-checking for production releases. + +--- + +## Text Summarization +**Definition:** The task of condensing a document into a shorter version that preserves the most important information, categorized as extractive (selecting existing sentences) or abstractive (generating novel text). + +### Theory & Explanation + +Extractive summarization selects existing sentences from the source document to form a summary. The classic approach is TextRank, an unsupervised graph-based algorithm: sentences are nodes, edge weights represent content overlap (typically cosine similarity using TF-IDF), and the PageRank algorithm iteratively computes sentence importance scores. The top-k highest-scoring sentences become the summary. Modern extractive approaches use BERT-based sentence scoring: encode each sentence in context of the full document, then classify each sentence as summary-worthy or not. BERT-based extractive summarization achieves ROUGE-1 F1 scores of 40-45 on CNN/DailyMail, significantly outperforming TextRank (ROUGE-1 ~30). The advantage of extractive summarization: guaranteed factual correctness and fluency (using the author's original phrasing). Disadvantage: no paraphrasing, sometimes lacks coherence when concatenating sentences from different parts of the document. + +Abstractive summarization generates novel sentences that may not appear in the source — it paraphrases, reorders, and compresses information. This is formulated as a sequence-to-sequence task: the input is the source document, the output is the summary. Key models: BART (Bidirectional and Auto-Regressive Transformer) is a denoising autoencoder: a bidirectional encoder + autoregressive decoder, pre-trained on text corruption tasks (token masking, text infilling, sentence permutation). T5 (Text-to-Text Transfer Transformer) frames all NLP tasks as text-to-text: input "summarize: [document]" → output "summary." PEGASUS pre-trains specifically for summarization by masking entire sentences in the input and predicting them as a pre-training objective, matching the summarization task structure. + +The main challenges for abstractive summarization are: (1) Factual consistency — models may hallucinate facts not present in the source ("The meeting decided to increase the budget" when the source said only "The meeting discussed the budget"). Up to 25-30% of abstractive summaries contain factual errors. (2) Repetition — models may repeat the same information. (3) Redundancy handling — the model must select the most important information from long documents. (4) Length control — generating summaries of exactly the desired length (50 words vs 200 words). Solutions: length penalties, coverage mechanisms (discourage attending to already-summarized content), and factual consistency verification using NLI models. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is the standard metric: ROUGE-N measures n-gram overlap, ROUGE-L measures longest common subsequence. + +### Example + +Summarizing a CNN news article (500 words → 3 sentences). Source: "The Federal Reserve raised interest rates by 25 basis points on Wednesday, marking the tenth consecutive increase since March 2022. The move was widely expected by economists, who predict at least one more rate hike this year. Fed Chair Jerome Powell said inflation, while still elevated, has shown signs of moderating..." Extractive summary (TextRank, top-3 sentences): "The Federal Reserve raised interest rates by 25 basis points..." + "The move was widely expected..." + "Fed Chair Jerome Powell said inflation..." — ROUGE-1: 0.52, factually correct but redundant. Abstractive summary (BART): "The Federal Reserve raised interest rates by 25 basis points for the tenth consecutive time since 2022. Economists expect at least one more hike this year as inflation shows signs of moderating." — ROUGE-1: 0.58, more concise and coherent but might omit some details. Human summary: "Fed raises rates again, signals possible pause as inflation eases." — 9 words, captures the key news angle (pause signal) that the automatic summaries missed. + +### Interview Questions + +**Q: How do you detect and prevent hallucination in abstractive summarization?** +A: Hallucination detection and prevention strategies: (1) Factual consistency verification — use an NLI model (e.g., BART-large-MNLI) to check if the source entails each summary sentence. If contradiction or neutral, flag as potentially hallucinated. (2) Entity grounding — extract all named entities from the source, verify every entity in the summary appears in the source. "Microsoft acquired GitHub" — check that both "Microsoft" and "GitHub" appear in the source. (3) Copy mechanisms — encourage the decoder to copy tokens from the source (Pointer-Generator networks, copy attention in BART). This is why BART's infilling pre-training helps — it learns to reconstruct masked spans from surrounding context, naturally preferring copying. (4) Confidence thresholds — hallucinated content often has lower generation probability; reject tokens with probability below a threshold (e.g., 0.3). (5) Contrastive learning — train the model to distinguish faithful summaries from hallucinated ones using contrastive examples. Despite these methods, state-of-the-art abstractive models still hallucinate 15-20% of the time on news articles and more on technical/scientific documents where jargon is hard to replicate. + +**Q: When would you choose extractive over abstractive summarization?** +A: Choose extractive summarization when: (1) Factual accuracy is critical — news headlines, legal document summaries, medical report summaries. Extractive summaries cannot introduce new facts; they use the author's exact words. (2) The source writing is already concise and well-structured (scientific papers, where abstractive models often change technical terminology in ways that lose precision). (3) You need to guarantee certain sentences are included (specific policy statements, key findings). (4) You have limited compute — extractive is faster, uses smaller models, and doesn't require autoregressive generation. Choose abstractive when: (1) Summary fluency and readability matter more — the model can combine ideas from different parts of the text into coherent prose. (2) Length compression is aggressive — abstractive summaries can be much shorter (10% of original) while extractive needs multiple sentences. (3) Source text is poorly written — abstractive can paraphrase and improve clarity. (4) Paraphrasing is desired — product reviews, where you want to avoid repetitive cut-and-paste. + +--- + +## Question Answering +**Definition:** The task of automatically answering questions posed in natural language, over a given context passage (reading comprehension) or over a knowledge base. + +### Theory & Explanation + +Extractive QA is the most common formulation: given a context passage and a question, predict the start and end positions of the answer span in the passage. BERT for QA (SQuAD 1.1) adds two linear layers on top of BERT's output: one for predicting the start token (S·T_i) and one for the end token (E·T_i), where T_i is BERT's representation of token i. The probability of span (i, j) is softmax(start at i) × softmax(end at j). The model is trained with cross-entropy loss on gold start and end positions. BERT-large achieves 90.9% F1 on SQuAD 1.1, beating human performance (91.2%) for the first time. However, this is on "answerable" questions with guaranteed spans — SQuAD 2.0 adds unanswerable questions, and models score ~86-89% F1, below human 89.5%. + +Abstractive QA generates the answer as free text, suitable for questions requiring synthesis or where the answer isn't a contiguous span. Models like T5, GPT-3, and Flan-T5 are fine-tuned on QA datasets (Natural Questions, TriviaQA) to generate answers directly: input = "question: [Q] context: [C]" → output = "[answer]." Abstractive QA can answer "Why did the stock market drop?" by synthesizing "The market dropped due to concerns about inflation and rising interest rates" from multiple passages. The challenge is hallucination — the model may generate plausible-sounding but incorrect answers. Abstractive QA is typically evaluated on exact match and F1 (treating prediction and reference as bags of tokens) as well as fluency and factual accuracy scores. + +Open-domain QA combines retrieval and reading: given a question, first retrieve relevant passages from a large corpus (Wikipedia, internal documents), then read those passages to extract/abstract the answer. The RAG (Retrieval-Augmented Generation) architecture uses a Dense Passage Retriever (DPR — a bi-encoder trained to retrieve relevant passages) to find top-k passages, then a BART decoder generates the answer conditioned on those passages. RAG achieves 44% exact match on Natural Questions (open-domain), significantly better than closed-book (model answering from memory alone, ~20%). The retriever-reader architecture is now the standard for production QA systems: the retriever handles the knowledge store, the reader handles comprehension, and both can be optimized independently. + +### Example + +Using BERT for extractive QA on SQuAD 2.0. Context: "Tesla was founded in July 2003 by Martin Eberhard and Marc Tarpenning. Elon Musk joined the company in 2004 as chairman of the board." Question: "Who founded Tesla?" BERT outputs: start logits: "Martin" = 8.2, "Elon" = 2.1, "Tesla" = -1.5. End logits: "Tarpenning" = 7.9, "Musk" = 1.8, "2003" = -0.5. Span (Martin Eberhard and Marc Tarpenning) has probability softmax(8.2) × softmax(7.9) ≈ 0.95. Question: "When was Tesla founded?" Start: "July" = 9.1, End: "2003" = 8.8. Span "July 2003" with probability 0.98. Unanswerable question: "Who is the CEO of Tesla?" The model computes P(answerable) = 0.15 — the question is about a fact not in the passage. SQuAD 2.0 requires the model to answer "no answer" when the answer is not present, which is done by comparing the best span score to a learned threshold. The model correctly predicts "no answer" for the CEO question. + +### Interview Questions + +**Q: How does BERT handle questions with no answer (SQuAD 2.0)?** +A: SQuAD 2.0 introduces unanswerable questions to test whether models refuse to answer when the answer isn't in the passage. BERT's approach: (1) During training, for unanswerable examples, the gold label is start=0, end=0 (the [CLS] token), indicating "no answer." The model learns to assign high start/end scores to [CLS] for unanswerable questions. (2) During inference, the model computes the best span probability (max over start × end, excluding [CLS]). If this probability is below a threshold (learned from the validation set), and the [CLS] span has a higher score than the best content span, predict "no answer." (3) The threshold is typically tuned on the dev set. For BERT-large on SQuAD 2.0, the null threshold is around 0.8-0.9 — if the best span score is less than threshold×[CLS] score, predict no answer. This achieves 86.8% F1. The main challenge: questions that are "almost answerable" — the passage contains related information but not the exact answer. + +**Q: What is the difference between reading comprehension and open-domain QA?** +A: Reading comprehension (RC) provides the context passage — the model's task is finding the answer within that passage. SQuAD, RACE, and TriviaQA (with provided context) are RC datasets. The problem is that the context is assumed to contain the answer, and the model simply needs to locate it. Open-domain QA (ODQA) provides only the question — the model must first retrieve relevant information from a large corpus, then read it. This is much harder because: (1) The retriever must find the right passage among millions, introducing retrieval errors (if the retriever misses the relevant passage, the model cannot answer). (2) Multiple passages may contain partial answers, requiring synthesis. (3) Evidence conflicts — different passages may contain contradictory information. The retriever-reader architecture (DPR + BERT, RAG) has separate components: the retriever is typically a bi-encoder (fast, embed all passages offline) and the reader is a cross-encoder (accurate but must process each passage independently). ODQA performance is bounded by both retrieval precision (top-k contains answer) and reader accuracy, so component improvements compound. + +--- + +## ALBERT +**Definition:** A Lite BERT — a parameter-efficient variant of BERT that reduces memory consumption while maintaining or improving performance through parameter sharing and factorized embedding parameterization. + +### Theory & Explanation + +ALBERT introduces two key parameter reduction techniques. Factorized embedding parameterization separates the vocabulary embedding size from the hidden layer size. In BERT, the embedding matrix E is of size V × H (vocabulary × hidden dimension, typically 30,000 × 768 = 23M parameters). ALBERT decomposes this into two matrices: V × E (vocabulary × small embedding dimension, e.g., 30,000 × 128 = 3.8M) and E × H (128 × 768 = 98K). This reduces embedding parameters by 6× (23M → 3.9M) without losing expressiveness because the hidden layer can still learn rich representations. The small embedding dimension is sufficient — it doesn't need to be as large as the hidden layer because the embedding space is constrained (one embedding per token type) while the hidden space handles rich contextual interactions. + +Cross-layer parameter sharing is the second technique: ALBERT shares all parameters (self-attention weights, feedforward weights, attention projections) across all 12 layers. The same W_Q, W_K, W_V, and feedforward layers are reused at every layer. This means the model has only one layer's worth of parameters, not 12. Total parameter reduction: ALBERT-base has 12M parameters vs BERT-base's 110M — a 9× reduction. Despite sharing parameters, each layer's output is different because the input to each layer is the preceding layer's output (which has been transformed by the shared parameters). This creates a form of deep iterative refinement: the same layer is applied 12 times, each time refining the representation. + +ALBERT also replaces BERT's Next Sentence Prediction (NSP) with Sentence Order Prediction (SOP). NSP was a binary classification task predicting whether two segments are consecutive from the same document or from different documents. This was too easy — the model could exploit topic shift and lexical overlap cues. SOP is harder: given two consecutive segments from the same document, predict whether they are in the correct order or swapped. This forces the model to learn discourse coherence and inter-sentence relationships. ALBERT outperforms BERT on GLUE, RACE, and SQuAD despite having 10× fewer parameters. ALBERT-xxlarge (12-layer shared, 4096 hidden, 64 attention heads) achieves state-of-the-art results on several benchmarks with 235M parameters (vs BERT-large's 340M). + +### Example + +Comparing ALBERT and BERT on the RACE reading comprehension benchmark (28K passages, 98K questions). BERT-base (110M params): 72.0% accuracy. ALBERT-base (12M params, same hidden size 768): 73.4% — 1.4% higher with 9× fewer parameters. ALBERT-large (18M params): 76.1%. ALBERT-xxlarge (235M params, 12 layers, each 4096 hidden): 89.4% — competitive with human performance and better than BERT-large (340M params, 24 layers depth). The key insight: ALBERT-xxlarge has 12 very wide layers (4096 hidden, 64 heads) rather than BERT-large's 24 narrower layers (1024 hidden, 16 heads). The width compensates for the lack of depth, and cross-layer parameter sharing prevents the width from exploding parameter count. Training ALBERT-xxlarge takes about the same time as BERT-large (due to similar hidden dimension throughput) but uses less memory. + +### Interview Questions + +**Q: Why does parameter sharing not hurt ALBERT's performance?** +A: Parameter sharing works well because: (1) Each layer's function is the same but the input differs — the first layer receives raw word embeddings, layer 2 receives refined representations, etc. The shared parameters learn to perform a general "refinement" operation that improves representations regardless of quality. (2) Stacking the same layer multiple times is equivalent to iteratively applying the same function f, creating a form of deep iterative refinement similar to optimization algorithms. (3) The shared layer receives gradient signals from all layers during backpropagation, providing richer training signal than individually trained layers — each layer gets gradients from its own output plus gradients from all higher layers. (4) ALBERT compensates for reduced depth by increasing width (hidden size). The parameter budget saved by sharing is redirected to wider hidden layers, and experiments show wider hidden layers + parameter sharing outperforms narrower layers without sharing. + +**Q: What are the limitations of ALBERT compared to BERT?** +A: ALBERT's limitations: (1) Training throughput — despite fewer parameters, ALBERT's training speed is similar to BERT because the computational cost is dominated by the hidden dimension operations, not the parameters. The same forward/backward pass through 12×768 layers costs similar to BERT's 12×768 layers — parameter sharing doesn't reduce FLOPs, only memory. (2) Inference latency — same as BERT for the same hidden size. The parameter savings primarily help memory and model storage, not speed. (3) Limited by width — ALBERT relies on very wide layers to compensate for depth, but width scales quadratically in attention (O(H²)) while depth scales linearly (O(L)). Very wide ALBERT (4096 hidden) has expensive attention (16M per head vs BERT's 1M). (4) Diminishing returns — ALBERT-xxlarge's improvements over ALBERT-large are modest, suggesting the design reaches a plateau. For tasks requiring deep hierarchical understanding (long-range dependencies), BERT's deeper architecture may have advantages. In practice, ALBERT is an excellent choice when model size or memory is constrained, while BERT is preferred when only raw accuracy matters and resources are abundant. + +--- + +## Perplexity (NLP) +**Definition:** A measurement of how well a language model predicts a sample of text, defined as exp(cross-entropy loss) — lower perplexity indicates better predictive performance. + +### Theory & Explanation + +Perplexity is the most common intrinsic evaluation metric for language models. It quantifies the model's uncertainty: perplexity of N means the model is as confused as if it had to choose uniformly among N alternatives at each prediction step. For a model that predicts each token given previous tokens, perplexity on a sequence of length T is: PPL = exp(-(1/T) × Σlog P(w_t | w_{ Account" even though there are no overlapping keywords. However, each (query, document) pair must be encoded together, making it infeasible to pre-compute document representations. + +The two-stage retrieval architecture is the standard production pattern: use a bi-encoder for fast candidate retrieval (return top-100 candidates from millions), then a cross-encoder for precise reranking (score and reorder the top-100). This combines the scalability of bi-encoders with the accuracy of cross-encoders. The bi-encoder retrieves recall (~90% of relevant docs in top-100), and the cross-encoder boosts precision (top-10 from the 100 candidates to 95%+ relevance). The two-stage approach is significantly faster than running a cross-encoder over the entire corpus (100 cross-encoder calls vs millions). This pattern powers production systems at Google Search, Bing, and most commercial semantic search engines. + +### Example + +Building a question-answering system for a 1M document knowledge base. With only a cross-encoder (BERT): each query-document pair takes 5ms. Searching all 1M documents serially would take 5,000 seconds (~83 minutes) — completely infeasible for real-time search. With a bi-encoder only (SBERT): pre-compute 1M document embeddings (offline, 2 hours). Query encoding: 10ms. Similarity search over 1M vectors via FAISS: 5ms. Total: 15ms per query. Recall@100: 88% (12% of relevant documents missed). With two-stage: bi-encoder retrieves top-100 candidates (15ms), cross-encoder reranks the 100 candidates (100 × 5ms = 500ms). Total: 515ms per query. Recall@10: 94%, Precision@10: 92% — much better than bi-encoder alone (Recall@10: 75%). The 500ms reranking cost is acceptable for interactive search, while the cross-encoder-only approach (83 minutes) is not. + +### Interview Questions + +**Q: When is a cross-encoder worth the extra cost over a bi-encoder?** +A: Use a cross-encoder when: (1) Accuracy requirements are strict — in legal discovery, medical literature search, or enterprise document retrieval where missing a relevant document or retrieving an irrelevant one has high cost. The cross-encoder captures nuanced relevance (negation, synonymy, paraphrasing, logical inference) that bi-encoders often miss. (2) The candidate pool is already small — reranking hundreds of items (top-100 candidate documents, 10 customer support responses, 20 recommended products) is fast and significantly better than bi-encoder-only. (3) Training data is limited — cross-encoders fine-tune effectively with few hundred labeled pairs because they directly learn the query-document interaction, while bi-encoders need more data to learn good representations. (4) Synonym and paraphrase handling is critical — cross-encoders can match "automobile" to "car" through pairwise attention even if never seen together during training. Use a bi-encoder when: query latency must be <100ms over millions of documents, or when you need to pre-compute embeddings for offline processing. The practical advice: start with a bi-encoder for recall, add cross-encoder reranking if precision is insufficient. + +**Q: How do you train a bi-encoder for retrieval?** +A: Bi-encoder training uses contrastive learning with positive and negative pairs. For each query, you need at least one relevant (positive) document and one or more irrelevant (negative) documents. The loss maximizes similarity for positive pairs and minimizes similarity for negative pairs. The standard loss is: L = -log(e^{sim(q,d+)/τ} / (e^{sim(q,d+)/τ} + Σe^{sim(q,d-)/τ})), where τ is a temperature hyperparameter (typically 0.01-0.1). Hard negative mining is critical: using random negatives is too easy, so training includes "hard negatives" — documents that are somewhat relevant but not the correct answer. In practice, in-batch negatives are used: for a batch of N (query, positive) pairs, the N-1 other positives in the batch serve as negatives for each query. This is efficient and provides a good distribution of negatives. After training, the bi-encoder is ready for vector indexing and retrieval. Additional data augmentation (back-translation of queries, query paraphrasing) can improve recall by 5-10%. diff --git a/techy/05-llm-rag-graph.md b/techy/05-llm-rag-graph.md new file mode 100644 index 0000000..ea62c71 --- /dev/null +++ b/techy/05-llm-rag-graph.md @@ -0,0 +1,1131 @@ +# 5. LLMs, RAG, PROMPT ENGINEERING, AGENTS, GRAPH ML, FINE-TUNING, EVALUATION + +## LLMs (Large Language Models) +**Definition:** Transformer-based neural networks with billions of parameters trained on internet-scale text via next-token prediction. Exhibit emergent abilities: reasoning, translation, code generation. + +### Theory & Explanation + +Large Language Models are deep neural networks built on the Transformer architecture, characterized by having billions (and more recently trillions) of parameters. They are trained through self-supervised learning on massive text corpora — essentially the entire public internet, books, academic papers, and code repositories. The core training objective is next-token prediction: given a sequence of tokens, the model learns to predict the most likely next token. This simple objective, scaled to trillions of training examples, forces the model to learn grammar, factual knowledge, reasoning patterns, translation, code syntax, and even some degree of world modeling. The pre-training phase is computationally massive — GPT-4 is estimated to have cost over $100 million in compute. After pre-training, models undergo instruction tuning and preference optimization (RLHF or DPO) to align with human preferences for helpfulness, honesty, and safety. + +Scaling laws, first formalized by Kaplan et al. (2020) from OpenAI and later refined by the Chinchilla scaling laws (Hoffmann et al., 2022), describe a power-law relationship between model performance and three factors: model size (parameters), dataset size (tokens seen during training), and compute budget (FLOPs). The key insight is that these three factors must be scaled together for optimality. The Chinchilla scaling law showed that for every doubling of model parameters, the training data should also double — a finding that led to training smaller models on more data (Chinchilla 70B outperformed Gopher 280B). Beyond a certain scale, emergent abilities appear that were not explicitly trained for: arithmetic reasoning, theory of mind, in-context learning, and chain-of-thought reasoning. These abilities only manifest at scale and cannot be predicted by extrapolating small-model performance curves. + +LLMs generate text auto-regressively: given a prompt, the model produces one token at a time, with each token conditioned on all previous tokens. The generation process has three key controls. Temperature (0.0 to 2.0) controls the randomness of token sampling — low temperature (0.1) makes output deterministic and focused, high temperature (0.8+) makes it creative and diverse. A temperature of 0.0 selects the most likely token every time (greedy decoding). Top-p (nucleus sampling) selects from the smallest set of tokens whose cumulative probability exceeds p — if p=0.9, the model only samples from tokens that together have 90% probability mass, cutting off the long tail of unlikely tokens. Top-k restricts sampling to the k most likely tokens at each step. These parameters are typically combined: for factual tasks, use low temperature (~0.1-0.3) and high top-p; for creative tasks, use temperature ~0.7-0.9. + +Context window size determines how many tokens the model can process at once. Early models like GPT-3 had 2K token context windows; modern models support 4K (GPT-3.5), 8K-32K (GPT-4), 128K (Claude 3, GPT-4 Turbo), 200K (Claude 3.5, Gemini 1.5 Pro), and even 1M-10M (Gemini 1.5 Pro experimental). Larger context windows enable processing entire documents, codebases, or hour-long videos. However, effective usage of long context remains challenging — models exhibit lost in the middle issues where information in the middle of the context is less reliably used than information at the beginning or end. The KV (Key-Value) cache is a critical optimization for inference: as the model generates tokens auto-regressively, it caches the key and value tensors from previous attention computations so they don't need to be recomputed for each new token. KV cache size grows linearly with sequence length and batch size, often becoming the primary memory bottleneck during generation — a 70B model generating 2K tokens might use 40GB+ just for KV cache. + +Quantization reduces model precision from FP16/BF16 to INT8 or INT4, reducing memory footprint 2-4x with minimal quality loss. Post-training quantization (PTQ) techniques like GPTQ, AWQ, and GGUF apply rounding and calibration to minimize error. Quantization enables running large models on consumer hardware — a 70B parameter model at 4-bit quantization fits in roughly 35GB of memory. Activation quantization is more challenging than weight quantization due to outliers. Inference engines like vLLM, TensorRT-LLM, and llama.cpp combine quantization with optimizations like continuous batching, PagedAttention (managing KV cache like virtual memory pages), and speculative decoding (using a small draft model to predict multiple tokens at once, verified by the large model in parallel). + +### Example + +GPT-4 with 1.7 trillion parameters and a 100K token context window. Zero-shot example: Prompt "Translate to French: Hello" generates "Bonjour" without any examples. Few-shot example: Provide 3 sentiment examples ("I love this movie: Positive", "This is terrible: Negative", "It was okay: Neutral") then prompt "The product is amazing:" — the model classifies as "Positive" by pattern matching the examples. Chain-of-Thought: On the GSM8K math benchmark, standard prompting achieves roughly 30% accuracy. Adding "Let's think step by step" before the answer (CoT prompting) improves accuracy to roughly 70% because the model generates intermediate reasoning steps before the final answer, reducing arithmetic errors. Temperature comparison: With temperature=0, multiple runs produce identical output; with temperature=0.8, the same prompt might produce different but equally valid responses. + +### Interview Questions + +**Q: What makes LLMs fundamentally different from smaller language models like BERT or traditional NLP models?** + +A: Three key differences. First, scale: LLMs have billions to trillions of parameters trained on internet-scale data, while BERT-based models have hundreds of millions. Second, emergent abilities: smaller models show no ability for in-context learning, chain-of-thought reasoning, or instruction following — these capabilities appear only above a threshold (roughly 1B+ parameters for basic ICL, 100B+ for advanced reasoning). Third, architecture: LLMs use decoder-only transformers optimized for generation (causal attention masks), while BERT uses encoder-only (bidirectional attention) optimized for understanding and classification. LLMs are few-shot learners by default; traditional models require task-specific fine-tuning. LLMs exhibit world knowledge compression — they learn facts, relationships, and reasoning patterns during pre-training; BERT learns representations useful for transfer learning but lacks generative capability. + +**Q: Explain temperature, top-p, and top-k for controlling text generation.** + +A: These parameters control the randomness and diversity of token sampling during autoregressive generation. Temperature (range 0-2, default ~1.0) scales the logits before softmax: temperature approaching 0 makes the highest-probability token dominant (greedy, deterministic output); higher temperatures flatten the probability distribution, making low-probability tokens more likely to be chosen (creative, diverse output). Temperature 0 is equivalent to always picking the argmax token. Top-k sampling restricts the next token selection to the k most likely tokens at each step, cutting off the long tail. For example, top-k=40 means the model only considers the 40 tokens with the highest probabilities. Top-p (nucleus sampling) is more adaptive: it selects the smallest set of tokens whose cumulative probability exceeds p. If p=0.9, the set grows for uniform distributions and shrinks for peaked ones. These are typically combined: top-k=40 first narrows candidates, then top-p=0.9 selects from within those, then temperature scaling is applied before sampling. For factual and QA tasks, use low temperature (0.1-0.3) and high top-p (0.95). For creative writing, use temperature 0.7-0.9 and top-p 0.9. + +**Q: What are scaling laws for LLMs and what do they imply?** + +A: Scaling laws describe power-law relationships between model performance and three factors: number of parameters (N), dataset size (D), and compute budget (C). The original Kaplan scaling law (2020) showed that performance improves predictably with scale — doubling parameters or data gives consistent log-linear improvement. The Chinchilla scaling law (2022) from DeepMind refined this: for compute-optimal training, model size and training tokens should be scaled equally — N and D should both double together. Previously, models were undertrained (too many parameters, not enough data), so Chinchilla 70B, trained on 1.4 trillion tokens, outperformed Gopher 280B trained on only 300B tokens. Implications: (1) Compute budget is the binding constraint — you trade off model size versus data size. (2) Smaller models trained on more data can match larger undertrained models. (3) Emergent abilities cannot be predicted from scaling curves of smaller models — they appear as phase transitions at specific scales. (4) The cost of training frontier models is dominated by compute, not data acquisition. + +### Related Concepts +Transformers, Pre-training, In-Context Learning, Fine-tuning + +--- + +## RAG (Retrieval-Augmented Generation) +**Definition:** Architecture combining document retrieval with LLM generation. Retrieves relevant context from knowledge base, injects into prompt, LLM generates grounded answer reducing hallucinations. + +### Theory & Explanation + +Retrieval-Augmented Generation addresses the fundamental limitation of LLMs: they have a fixed knowledge cutoff (the date their training data ended), they cannot access private or proprietary information, and they hallucinate when asked about unfamiliar topics. RAG solves this by connecting the LLM to an external knowledge base — documents, databases, APIs — at inference time. The core pipeline: user query comes in, optionally gets rewritten or decomposed, a retriever searches the knowledge base for relevant documents, those documents are ranked and injected into the LLM's context window, and the LLM generates an answer grounded in the retrieved context. Because the LLM can cite sources and is constrained to answer based on provided context, hallucinations are dramatically reduced. RAG also enables updating knowledge without retraining — just add or modify documents in the knowledge base. + +The retrieval stage is the most critical component and typically uses hybrid search combining dense embeddings and sparse (BM25) retrieval. Dense retrieval: documents are split into chunks (typically 256-1024 tokens), each chunk is embedded into a high-dimensional vector (768-3072 dimensions) using an embedding model like OpenAI ada-002, Cohere embed-v3, or open-source models like bge-large, E5, or GTE. The query is embedded with the same model, and cosine similarity or dot product finds the nearest chunks. BM25 provides lexical matching — it is a bag-of-words ranking function that handles exact keyword matches and term frequency. Hybrid search combines both using Reciprocal Rank Fusion (RRF): RRF score = sum(1/(k + rank_i)) for each result in each ranking, merging results into a final combined ranking. This is robust because dense retrieval captures semantic similarity (what is the policy on working from home matches remote work policy), while BM25 captures exact terms (refund policy matches documents containing those exact words). + +Chunking strategy significantly impacts retrieval quality. Fixed-size chunking splits at token boundaries (e.g., every 512 tokens with 128-token overlap) — simple but can split sentences mid-thought. Semantic chunking uses embedding similarity to detect topic boundaries, creating chunks aligned with topical shifts. Recursive chunking applies multiple levels: small chunks (256 tokens) for granular precision, with parent chunks (2048 tokens) providing surrounding context when a small chunk is retrieved — the parent retriever pattern retrieves small chunks but returns their parent documents for the LLM context, balancing precision with context completeness. Document structure (headings, sections, list items) provides natural chunk boundaries. Metadata (document title, date, section, page number) is attached to each chunk for filtering and source citation. + +Vector databases (Pinecone, Chroma, Weaviate, Qdrant, Milvus) store embeddings and provide Approximate Nearest Neighbor (ANN) search. The standard ANN algorithm is Hierarchical Navigable Small Worlds (HNSW), which builds a multi-layer graph structure: lower layers contain more nodes and provide fine-grained navigation, higher layers are sparser for coarse search. HNSW offers O(log n) search complexity with high recall. Production vector databases add: metadata filtering (filter chunks by date, source, category before vector search), scalar quantization (compressing float32 vectors to int8 for 4x memory reduction), partitioning and namespacing (separate indexes for different clients or data types), and serverless architectures (Pinecone serverless, Chroma Cloud) that decouple storage from compute for cost efficiency at scale. The choice of embedding model is critical — domain-specific embedding models (e.g., Legal-BERT, BioBERT) significantly outperform general models on specialized retrieval tasks. + +The generation stage deserves equal attention. The classic RAG pattern simply concatenates retrieved chunks before the user query in the prompt. More sophisticated approaches include: (1) Re-ranking using a cross-encoder (e.g., Cohere rerank-v3, BGE-reranker) that jointly scores query-chunk pairs — cross-encoders are more accurate than bi-encoders but cannot scale to millions of documents, so they are used as a second-stage filter on the top 50-100 results from the bi-encoder. (2) Query rewriting: before retrieval, the system rewrites the user's query to be more search-friendly (e.g., expanding acronyms, adding synonyms, decomposing complex questions into sub-questions). (3) HyDE (Hypothetical Document Embeddings): generate a hypothetical ideal document from the query, embed that, and use it for retrieval — effective when the query is short and the embedding space is better populated by document-like text. (4) Self-RAG: the LLM generates retrieval decisions — when to retrieve, whether retrieved passages are relevant, and whether the generation is supported by them. This creates a dynamic retrieval loop rather than a single retrieval step. + +Advanced RAG variants address specific failure modes. Corrective RAG (CRAG) adds a relevance evaluator: if retrieved documents score below a threshold, the system triggers web search to find better sources. Agentic RAG treats retrieval as a tool call within an agent loop — the LLM can decide to retrieve, search the web, perform calculations, or ask clarifying questions before answering. GraphRAG (from Microsoft) builds a knowledge graph from documents by extracting entities and relationships, then uses community detection and summarization for global sensemaking queries. Self-RAG trains the LLM to output special tokens indicating retrieval needs and passage relevance, improving answer faithfulness. The RAPTOR model builds a hierarchical summary tree over document chunks, enabling retrieval at multiple abstraction levels — a query about a general topic retrieves a high-level summary, while a specific question drills into leaf-level chunks. + +Common RAG failure modes and their fixes include: (1) Irrelevant retrieval: solution — query rewriting, HyDE, or adding a reranker that filters irrelevant passages before the LLM sees them. (2) Missing context (the answer is not in the retrieved chunks): solution — increase top-k, adjust chunk size, use sliding window chunks with overlap, or implement the parent retriever to provide broader context. (3) LLM ignores provided context: solution — strengthen the system prompt to emphasize context-only answering, use structured prompts with clear context and query separation, or fine-tune the model to follow context instructions (instruction-tuned models are less prone to this). (4) Lost in the middle (relevant chunk in the middle of context): solution — place the most relevant chunk last (recency bias) or first (primacy bias), or reorder chunks by relevance in descending order. + +### Example + +Company policy Q&A system: 500 PDFs of HR and IT policies are chunked into 100,000 chunks (512 tokens each, 128-token overlap). Each chunk is embedded with OpenAI ada-002 (1536 dimensions) and stored in a Pinecone serverless index. User asks: "What is our remote work policy?" The query is first rewritten by a small LLM to "remote work policy and WFH guidelines 2025." Hybrid search retrieves the top 20 chunks (10 via dense cosine similarity, 10 via BM25 lexical match). These are fused with RRF (k=60) and the top 5 are passed to a cross-encoder reranker (BGE-reranker-v2-m3). The top 3 reranked chunks are injected into the GPT-4 prompt. The final prompt is: "You are a helpful assistant. Answer the question based ONLY on the following context. If the context does not contain the answer, say you do not know. Context: [chunk1] [chunk2] [chunk3]. Question: What is our remote work policy?" GPT-4 generates a concise answer citing specific policy sections and page numbers from the PDFs. No hallucination is possible because the answer must cite the provided context. + +### Interview Questions + +**Q: RAG vs Fine-tuning vs Prompt Engineering — when to use each?** + +A: RAG is for knowledge access — use when the LLM needs external or up-to-date information that it was not trained on, such as company policies, product documentation, or recent news. Fine-tuning is for behavior and style adaptation — use when you need consistent output format, tone, or domain-specific terminology that prompting alone cannot achieve, such as writing in a specific legal language, following a strict schema, or adapting to a specialized vocabulary. Prompt engineering is for quick task adaptation — use for simple formatting, role-playing, or instruction-following where few-shot examples suffice. In practice, these are used together: RAG provides factual grounding, prompt engineering structures the interaction, and fine-tuning handles specialized output requirements. RAG wins for evolving knowledge (no retraining needed), fine-tuning wins for deep behavioral changes, and prompt engineering wins for rapid iteration. + +**Q: How to evaluate RAG system quality?** + +A: RAG evaluation covers both retrieval quality and generation quality. Retrieval metrics: hit rate (percentage of queries where at least one relevant chunk is in the top-k), mean reciprocal rank (MRR — rank position of the first relevant result), normalized discounted cumulative gain (nDCG — accounts for graded relevance), and precision at k. Generation metrics from the RAGAS framework: faithfulness (does the answer stay true to the retrieved context — measured by decomposing the answer into claims and verifying each against the context), answer relevancy (does the answer address the question), context precision (are the retrieved chunks relevant to the question), and context recall (are all necessary chunks retrieved). Additional evaluation: human preference ratings, LLM-as-judge scoring with GPT-4 or Claude as evaluator, and A/B testing on live user traffic. A production RAG system should have automated evaluation pipelines that run on every knowledge base update. + +**Q: Common RAG failures and how to fix them?** + +A: Four major failure modes exist. (1) Irrelevant retrieval — the retrieved chunks are not about the query. Fixes: query rewriting (expand the query with synonyms and context), HyDE (embed a hypothetical ideal answer instead of the query), add a cross-encoder reranker, improve chunk quality with semantic chunking. (2) Missing context — the answer exists in the knowledge base but was not retrieved. Fixes: increase top-k, reduce chunk size to improve precision, use sliding window chunks with overlap, implement parent retriever for broader coverage, improve embedding model quality. (3) LLM ignores context — the model generates answers from its pre-training knowledge instead of the provided context. Fixes: strengthen system prompt (emphasize context-only answering with explicit instructions), use structured prompt format with clear separation between context and query, fine-tune the model on context-following tasks, use instruction-tuned models. (4) Lost in the middle — relevant chunk is in the middle of the context and the model ignores it. Fixes: reorder chunks by relevance (most relevant last or first), reduce total context length, use models with better long-context attention mechanisms. + +### Related Concepts +LLMs, Vector Databases, Embeddings, Fine-tuning + +--- + +## Prompt Engineering +**Definition:** Designing input prompts to get desired outputs from LLMs. Critical skill for effective LLM use. + +### Theory & Explanation + +Prompt engineering is the practice of carefully crafting inputs to large language models to elicit desired behaviors, formats, and reasoning patterns. It has emerged as a critical skill because LLMs are highly sensitive to phrasing, structure, and the examples provided in the prompt. A well-engineered prompt can dramatically improve output quality — sometimes doubling accuracy on reasoning tasks — while a poorly designed prompt leads to irrelevant, incorrect, or poorly structured responses. The fundamental components are the system prompt (sets the role, persona, rules, and constraints for the model's behavior throughout the conversation) and the user prompt (the specific query or task). Separating these allows fine-grained control: the system prompt establishes persistent guardrails while the user prompt handles the specific request. + +Zero-shot prompting gives the model a task without examples. It relies entirely on the model's pre-training and instruction-following ability. This works well for common tasks (translation, summarization, simple classification) but fails for nuanced or domain-specific tasks. Few-shot prompting provides 2-5 examples of input-output pairs before the target query, leveraging the model's in-context learning ability. The examples serve as implicit instructions about format, reasoning style, edge cases, and output structure. Few-shot is more reliable but costs more tokens. Key considerations for few-shot: examples should cover edge cases (not just the most common patterns), be representative of the task distribution, and be ordered from simple to complex. Chain-of-Thought (CoT) prompting instructs the model to output its reasoning process step by step before the final answer. This was introduced by Wei et al. (2022) and dramatically improves performance on arithmetic, common sense, and symbolic reasoning tasks. CoT works by offloading intermediate computation into the output tokens, effectively giving the model more compute at test time. Accuracy on math benchmarks improves from roughly 30% to 70% with CoT. Variants include zero-shot CoT (just add "Let's think step by step"), few-shot CoT (provide reasoning examples), and tree-of-thought (explore multiple reasoning paths simultaneously). + +Structured output prompting forces the model to generate parseable formats like JSON, XML, or Markdown tables. Best practices include: specifying the exact schema in the prompt, providing an example output, using delimiters to separate sections clearly, and instructing the model not to include markdown code fences around JSON. For mission-critical parsing, constrained decoding techniques like grammar-guided generation (Outlines library, lm-format-enforcer) or JSON-mode APIs (OpenAI JSON mode, Gemini response_schema) enforce the output structure at the token probability level, guaranteeing valid JSON. Without these, LLMs may occasionally produce malformed JSON with trailing commas, missing brackets, or extra text. + +Advanced prompting techniques include: (1) ReAct (Reasoning + Acting) — interleave reasoning steps with tool calls, enabling agents to gather information and act upon it. The format is Thought: ... Action: tool_name(input) ... Observation: ... Final: ... (2) Self-Consistency — generate multiple outputs with high temperature, then take the majority vote or most consistent answer. Improves reliability by 5-15% on reasoning tasks. (3) Chain-of-Note — generate retrieval notes before answering in RAG, improving answer quality. (4) Structured Chain-of-Thought — force the model to fill in pre-defined reasoning steps rather than free-form reasoning. (5) Constitutional AI — provide the model with principles and rules it must follow, used for safety alignment without human feedback. (6) Negative prompting — explicitly telling the model what NOT to do, often as effective as positive instructions. For example: "Do not include any information that is not in the provided context. Do not speculate. Do not give medical advice." + +Best practices for prompt engineering include: be specific and precise about the desired output, put instructions at the beginning of the prompt (primacy effect), use delimiters to separate sections (triple quotes, XML tags, markdown headers), provide few-shot examples for format and edge cases, ask the model to reason step by step for complex tasks, limit scope to a single task per interaction, use persona and role-setting to establish context, and iterate systematically (change one variable at a time, track performance). The prompt should be treated as code — version controlled, tested, and optimized through systematic experimentation. + +### Example + +System prompt: "You are a data science tutor specializing in SQL. Your role is to help users write SQL queries given natural language questions and database schemas. Only use tables and columns from the provided schema. If a query cannot be written with the available schema, explain why. Format your response as: Explanation: [step-by-step reasoning], SQL: [the generated query]." + +Few-shot examples: Provide 2 examples of question-to-SQL pairs. Example 1: Schema: users(id, name, email, signup_date). Question: "How many users signed up in 2024?" SQL: "SELECT COUNT(*) FROM users WHERE signup_date BETWEEN '2024-01-01' AND '2024-12-31';" Example 2: Same schema. Question: "List all users who signed up in the last 30 days." SQL: "SELECT name, email FROM users WHERE signup_date >= DATE_SUB(CURRENT_DATE, INTERVAL 30 DAY);" + +User prompt: "Schema: {schema}. Question: {question}" + +The CoT version adds: "Let's think step by step. First, identify the tables needed. Second, identify the columns. Third, identify filtering conditions. Fourth, construct the query." This structured reasoning reduces column name hallucinations and join errors by roughly 60% compared to direct generation. + +### Interview Questions + +**Q: What is Chain-of-Thought prompting and why does it work?** + +A: Chain-of-Thought prompting instructs the LLM to output intermediate reasoning steps before arriving at a final answer. It works because: (1) It offloads computation into the output tokens — autoregressive generation effectively gives the model more compute for reasoning by using each output token as an intermediate computation step. (2) It breaks complex problems into manageable sub-steps, reducing the cognitive load on a single attention pass. (3) It creates a verifiable reasoning chain — if the final answer is wrong, you can identify which step failed. (4) It aligns with how training data is structured (many reasoning traces exist in pre-training text). On the GSM8K math benchmark, standard prompting achieves roughly 30% accuracy. Zero-shot CoT (adding "Let's think step by step") improves to roughly 70%. Few-shot CoT (providing 8 reasoning examples) reaches roughly 92% with GPT-4. CoT is most effective for math, logic, coding, and multi-step reasoning tasks. It provides little benefit for simple factual recall or classification tasks. + +**Q: Zero-shot vs Few-shot prompting — when to use each?** + +A: Zero-shot prompting should be used when the task is common and well-represented in the model's training data (translation between major languages, summarization of news articles, general question answering, simple sentiment analysis), when token costs or latency constraints limit prompt length, or when iterating quickly on prompt design. Few-shot prompting should be used for domain-specific tasks (legal document classification, medical terminology extraction, custom data formats), when output format consistency is critical (structured JSON, specific XML schemas), when the task has edge cases that need explicit handling, or when zero-shot performance is below acceptable thresholds. Few-shot is generally more reliable because examples serve as implicit instructions — a model might misinterpret "extract the date" in zero-shot (which date? which format?) but two examples make it unambiguous. The trade-off is token cost: each example adds tokens to every request. A good practice is to start with few-shot, then distill the examples into the instruction text if they become stable, gradually moving toward zero-shot for efficiency. + +**Q: How to structure a prompt for reliable structured output?** + +A: For reliable structured output (typically JSON), follow these principles: (1) Define the exact schema in the prompt — specify each field name, type, allowed values, and nesting structure. (2) Provide one complete example output with correct formatting. (3) Use a system prompt to set strict format rules: "Always respond with valid JSON only. No markdown, no code fences, no explanation outside the JSON." (4) Use delimiters to separate the schema definition, example, and actual input. (5) For production systems, use constrained decoding — OpenAI's JSON mode, Gemini's response_schema parameter, or libraries like Outlines or lm-format-enforcer that enforce the grammar at generation time. (6) Always validate and parse the output server-side with a JSON parser, with error handling for malformed responses. (7) In few-shot, include both typical and edge cases. Example: System: "You are a JSON generator. Output ONLY valid JSON according to the schema." User: "Schema: {name: string, age: number, email: string}. Example: {name: 'John Doe', age: 30, email: 'john@example.com'}. Now output JSON for: {input}." + +### Related Concepts +LLMs, In-Context Learning, Agents + +--- + +## Agents & Tool Use +**Definition:** LLM-powered systems that perceive, reason, use tools (APIs, search, code), and act autonomously via ReAct loop. + +### Theory & Explanation + +An LLM agent is an autonomous system that combines a language model with tools, memory, and planning capabilities to accomplish complex tasks that require multiple steps, external information, and interaction with the world. Unlike a standard LLM that passively generates text from a prompt, an agent actively decides what to do next based on observations from its environment. The core loop, popularized by the ReAct (Reasoning + Acting) pattern, is: (1) Thought — the LLM reasons about the current state and decides what needs to be done. (2) Action — the LLM outputs a structured command to invoke a tool, such as searching the web, calling an API, executing code, or querying a database. (3) Observation — the tool's output is fed back into the LLM's context. (4) Repeat — the LLM processes the observation, generates a new thought, and decides on the next action. (5) Final — when the task is complete, the LLM outputs the final answer. Each cycle extends the context window, providing a running trace of the agent's reasoning and actions. + +Tools are the bridge between the LLM and the external world. A tool is defined by its name, description, input parameters (JSON schema), and output format. The LLM uses function calling — it outputs a JSON object specifying which tool to call and with what parameters. The system intercepts this JSON, executes the tool, and returns the result as an observation. Common tool categories include: (1) Information retrieval — web search, vector database query, document loaders. (2) Computation — Python interpreter, calculator, symbolic math engine. (3) APIs — Slack, email, CRM, calendar, GitHub, database query. (4) Code execution — sandboxed Python/JavaScript runtime for code generation tasks. (5) Memory — store and retrieve information across sessions. The tool description is critical for correct selection — the LLM uses it to decide which tool matches the current need. A vague description leads to wrong tool selection; a precise description with usage examples improves accuracy. + +Memory in agents operates at multiple levels: (1) Short-term memory — the conversation history within the current context window. This is limited by the model's context length but provides rich, detailed information about recent actions. (2) Long-term memory — external storage (vector database, key-value store, SQL database) where the agent can save and retrieve information across sessions. This is essential for personalized agents that need to remember user preferences, past interactions, and ongoing tasks. (3) Episodic memory — records of past agent runs, including successful and failed strategies, which the agent can learn from. (4) Procedural memory — stored tool definitions and action patterns. Managing context window limits is a major engineering challenge: agents that run for many steps eventually fill the context. Solutions include summarization (compress older turns), sliding windows (drop oldest turns), and retrieval-augmented memory (store past turns in a vector DB and retrieve relevant ones). + +Planning enables agents to handle complex, multi-step tasks that cannot be completed in a single action cycle. (1) Task decomposition — break a complex request into sub-tasks. For example, "Book a flight to New York" decomposes into search flights, compare options, select flight, enter passenger details, process payment, confirm booking. (2) Planning with ReAct — the agent discovers the plan dynamically by reasoning about each step. (3) Plan-ahead — the agent generates a complete plan before executing, then executes step by step. (4) Tree-of-Thought — the agent explores multiple planning paths simultaneously, pruning dead ends. (5) Replanning — when an action fails or produces unexpected results, the agent updates its plan. The Plan-and-Solve pattern (Wang et al. 2023) improves over ReAct by generating a high-level plan first, then executing it step by step, reducing backtracking and error recovery costs. + +Multi-agent systems coordinate multiple specialized agents to solve complex problems. Common patterns include: (1) Orchestrator + Workers — a supervisor agent decomposes tasks and delegates to specialized worker agents (researcher, coder, reviewer). (2) Debate — multiple agents with different perspectives discuss a problem and converge on an answer, improving correctness. (3) Hierarchical — agents at different levels of abstraction (strategic, tactical, operational). Popular frameworks include LangGraph (state-machine-based agent orchestration with cycles, conditional edges, and persistent state), AutoGen (Microsoft's multi-agent conversation framework), and CrewAI (role-based agent teams with task delegation). LangGraph is the most flexible, allowing developers to define agents as nodes in a directed graph with explicit state management — this enables complex patterns like human-in-the-loop, parallel agent execution, and dynamic tool assignment. + +Safety and guardrails are essential for production agent systems. Key risks include: (1) Tool abuse — the agent calling expensive APIs in infinite loops. Mitigation: rate limiting, budget tracking per session, max step limits, timeout per tool call. (2) Prompt injection — user input that hijacks the agent's system prompt. Mitigation: input sanitization, separating user input from instructions, using different models for different trust levels. (3) Hallucination propagation — a hallucinated observation leading to further hallucinated reasoning. Mitigation: verification steps, confidence thresholds, human approval for high-stakes actions. (4) Data leakage — the agent writing sensitive information to external tools. Mitigation: output filtering, data classification labels, allowlists for tool outputs. Guardrails frameworks like NVIDIA NeMo Guardrails, Guardrails AI, and LangChain's Guardrails provide programmable safety rules that intercept and validate agent actions before execution. + +### Example + +Customer support agent for an e-commerce company. User sends a message: "Where is my order #ORD12345? It was supposed to arrive yesterday." The agent begins its ReAct loop. Thought: "The user is asking about their order status. I need to retrieve order information. I will use the get_order function with the order ID." Action: get_order(order_id="ORD12345"). Observation: returns {status: "shipped", carrier: "FedEx", tracking: "FX123456789", estimated_delivery: "2025-07-21", current_location: "Memphis, TN hub", last_scan: "2025-07-20 06:30 AM"}. Thought: "The package is still in transit at the Memphis hub. It is one day late. I should inform the user and offer options — they may want to contact the carrier or get a refund for shipping." Action: get_shipping_policy(carrier="FedEx", delay_days=1). Observation: returns {policy: "FedEx guarantees Ground delivery within 5 business days. Late deliveries qualify for shipping fee refund on request."} Final thought: "I now have enough information to respond to the user." Final response: "Your order ORD12345 is currently at the FedEx Memphis hub and is running one day late. The new estimated delivery is tomorrow, July 21. Since it is past the original delivery date, you are eligible for a shipping fee refund per our carrier policy. Would you like me to initiate the refund request? You can also track the package in real-time here: fedex.com/track/FX123456789." + +If the user then asks "Yes, please refund shipping," the agent loops again: Thought: "User wants a shipping refund. I need to initiate a refund request. The get_order showed the payment method was Visa ending in 4242." Action: initiate_shipping_refund(order_id="ORD12345", reason="delayed_delivery"). Observation: returns {refund_id: "RF9876", status: "approved", amount: "$12.99", estimated_credit: "3-5 business days"}. Final: "I have initiated a $12.99 shipping refund. It will be credited to your Visa ending in 4242 within 3-5 business days. Your refund ID is RF9876 for reference." + +### Interview Questions + +**Q: Difference between RAG and an Agent?** + +A: RAG is a passive retrieval system — it retrieves information once based on the query and injects it into the prompt before the LLM generates a response. The LLM does not decide whether or when to retrieve; retrieval is a fixed preprocessing step. An agent is an active decision-maker — it decides what to do, when to use tools, how many times to loop, and when to stop. Agents can retrieve information, then based on that information decide to retrieve more, perform a calculation, call an API, or ask a clarifying question. RAG is a single-turn pattern (query in, answer out). Agents are multi-turn loops (perceive, reason, act, observe, repeat). RAG is appropriate when the task is straightforward factual Q&A. Agents are necessary when the task requires multiple steps, conditional logic, tool orchestration, or dynamic information gathering. In practice, many production systems use both: an agent that has retrieval as one of its tools. + +**Q: What is the ReAct agent pattern?** + +A: ReAct (Reasoning + Acting) is a prompting framework that interleaves reasoning traces with action steps. The pattern is: Thought (what is the current state and what should I do next), Action (call a tool with specific parameters), Observation (tool output is fed back), repeat until the task is complete, then Final (output the answer). The key insight is that reasoning traces help the model keep track of progress, recover from errors, and generalize to novel situations. Without explicit reasoning (just action-observation loops), the model loses context and makes more errors. With reasoning traces, the model can correct itself mid-task. ReAct was shown to outperform both chain-of-thought (reasoning only, no actions) and Act-only (actions without reasoning) on knowledge-intensive tasks. In LangGraph, ReAct is implemented as a state graph where each node is either a reasoning step, a tool call, or a final response, with conditional edges determined by the LLM's output. + +**Q: How do you ensure agent safety and prevent harmful actions?** + +A: Agent safety requires multiple layers of guardrails. (1) Scope limitation: restrict available tools to only what the agent needs — do not give an agent access to production database writes, billing APIs, or user password resets unless absolutely necessary. (2) Rate limiting and budget tracking: cap the number of tool calls per session (e.g., max 20 steps), set monetary limits for paid APIs, implement per-minute rate limits. (3) Human-in-the-loop: require human approval for high-risk actions — any action that modifies data, sends messages, spends money, or affects other users should go through a confirmation step. (4) Input validation: sanitize user inputs to prevent prompt injection where malicious input hijacks the agent's instructions. (5) Output filtering: screen generated responses for PII, toxic content, or policy violations before delivery. (6) Timeouts: terminate agent loops that exceed time limits. (7) Monitoring and auditing: log all agent actions, tool calls, and decisions for post-hoc analysis. (8) Determinate state: design the agent to be safe even if it gets stuck in a loop — stateless operations and idempotent tool calls where possible. + +### Related Concepts +LLMs, RAG, Function Calling, ReAct + +--- + +## Graph-Based ML & Knowledge Graphs +**Definition:** Models operating on graph data (nodes=entities, edges=relationships). Used for recommendation, fraud, drug discovery, multi-hop reasoning. + +### Theory & Explanation + +Graph-based machine learning operates on data structured as graphs — collections of nodes (vertices) connected by edges (relationships). Unlike Euclidean data (images, text, audio) where each sample has a fixed grid of neighbors, graph data is irregular: each node can have a variable number of neighbors arranged in arbitrary topology. This fundamental difference makes standard deep learning architectures (CNNs, RNNs, Transformers) inapplicable to graph data without modification. Graph Neural Networks (GNNs) were developed to handle this irregular structure. Key graph concepts: a graph G = (V, E) where V is the node set and E is the edge set. Nodes can have features (attribute vectors), edges can have features (relationship type, weight, direction), and the graph can be directed, undirected, or heterogeneous (multiple node and edge types). The adjacency matrix A encodes connections: A[i][j] = 1 if there is an edge from node i to node j. Node degree is the number of edges incident to a node. Homophily is the tendency of connected nodes to have similar features or labels (social networks: friends share interests). Heterophily is the opposite — connected nodes differ (fraud networks: fraudsters connect to legitimate accounts). + +Knowledge Graphs (KGs) are a specific type of graph representing facts as triples: (head entity, relation, tail entity). For example: (Einstein, born_in, Germany), (Germany, located_in, Europe). KGs are used for question answering, recommendation, and reasoning. Querying KGs uses SPARQL (a graph query language similar to SQL for graphs) or Cypher (Neo4j's declarative graph query language). Cypher example: MATCH (p:Person)-[:BORN_IN]->(c:Country)-[:LOCATED_IN]->(cont:Continent) WHERE cont.name = 'Europe' RETURN p.name. This finds all people born in European countries — a two-hop path. KGs built from text typically use named entity recognition (NER) to extract entities and relationship extraction to identify connections. Popular KGs include Wikidata (90M+ entities), DBpedia (extracted from Wikipedia), and domain-specific KGs like UMLS (medical), GeneOntology (biology), and Freebase. + +Graph Neural Networks learn node representations by aggregating information from neighbors — this is the message passing framework. In each layer, each node: (1) gathers messages from its neighbors (typically transformed via learned weight matrices). (2) Aggregates those messages (sum, mean, max, or attention-weighted sum). (3) Updates its own representation by combining its previous representation with the aggregated message. After L layers, a node's representation encodes information from its L-hop neighborhood. Key GNN architectures differ in how they perform message passing and aggregation. Graph Convolutional Networks (GCN, Kipf & Welling 2017) apply a normalized adjacency-based convolution: each neighbor's contribution is weighted equally (by degree normalization). Graph Attention Networks (GAT, Velickovic et al. 2018) learn attention weights for each neighbor, allowing the model to focus on more important neighbors — critical for graphs where neighbor relevance varies. GraphSAGE (Hamilton et al. 2017) uses inductive sampling: instead of using all neighbors, it samples a fixed-size neighborhood, enabling scaling to billion-node graphs and generalization to unseen nodes. + +GraphRAG is a recently popularized approach from Microsoft that combines knowledge graphs with RAG. The pipeline: (1) Extract entities and relationships from documents using an LLM. (2) Build a knowledge graph connecting these entities. (3) Apply community detection algorithms (Leiden, Louvain) to find groups of closely related entities. (4) Summarize each community using an LLM, producing natural language descriptions of what each community represents. (5) For local questions (specific facts), traverse the KG to find relevant entities. For global questions (trends, themes, summaries), use community summaries. GraphRAG excels at multi-hop reasoning questions that standard RAG struggles with. For example, "How does the company's approach to data privacy affect its cloud storage product roadmap?" requires connecting privacy policies to product features across multiple documents — standard RAG with chunk retrieval may miss these connections, while GraphRAG explicitly models the relationships. GraphRAG is more computationally expensive (requires building and maintaining the KG) but provides superior performance on global sensemaking and multi-hop questions. + +Applications of graph ML span many domains. Recommendation systems: nodes are users and items, edges represent interactions (purchases, views, likes). GNNs learn user and item embeddings by propagating information through the interaction graph, capturing collaborative filtering signals. Fraud detection: nodes are accounts, transactions, devices, IP addresses. Fraud patterns often involve sub-graph structures (dense money transfer rings, accounts sharing devices). GNNs can flag suspicious nodes by detecting anomalous neighborhood patterns. Drug discovery: molecules are graphs where atoms are nodes and bonds are edges. GNNs predict molecular properties, drug-target interactions, and toxicity. Social network analysis: community detection, influence propagation, link prediction (predicting future friendships or collaborations). Traffic forecasting: road networks are graphs, GNNs predict traffic flow by modeling spatial dependencies between road segments. + +### Example + +Fraud detection in financial transactions: The graph has three node types — accounts (100M nodes), transactions (1B nodes), and devices (50M nodes). Edges: account-to-transaction (sent/received), account-to-device (logged_in_from). Each node has features: account has balance, age, transaction history features; transaction has amount, timestamp, location; device has IP, browser fingerprint. A 3-layer GAT network is trained on labeled historical fraud data. Message passing in the first layer aggregates features from connected transactions to an account — an account receiving many small transactions then sending one large outgoing transaction gets flagged. The second layer aggregates across devices — if multiple flagged accounts share the same device, that device becomes high-risk. The third layer propagates device risk to all accounts using that device. GAT's attention mechanism learns that transaction amount and frequency are more important than location for fraud detection. The model achieves 0.95 AUC-ROC, catching 85% of fraud with a 0.1% false positive rate. Link prediction variant: predict whether an account will send money to another account — used for early fraud detection before the transaction completes. + +### Interview Questions + +**Q: What is GraphRAG and how does it differ from standard RAG?** + +A: GraphRAG (Microsoft, 2024) extends standard RAG by building a knowledge graph from the document corpus. The process involves: extracting entities and relationships using an LLM, building a KG, running community detection to find clusters of related entities, summarizing each community with an LLM, and using both the KG (for local/factual questions) and community summaries (for global/abstract questions) at retrieval time. Standard RAG retrieves text chunks via embedding similarity, which works well for single-hop factual questions but struggles with multi-hop questions (connecting information across documents) and global sensemaking questions (trends, themes, synthesis). GraphRAG addresses these by explicitly modeling entity relationships, enabling traversal across multiple hops, and providing community-level summaries for holistic questions. The trade-off: GraphRAG is significantly more expensive to build and maintain (requires LLM calls for entity extraction and community summarization), and it adds latency. Use standard RAG for simple Q&A on well-structured documents, GraphRAG for complex multi-hop questions and analysis across many documents. + +**Q: GCN vs GAT — when to use each?** + +A: GCN (Graph Convolutional Network) applies a fixed, normalized aggregation where each neighbor contributes equally (weighted by degree normalization). GAT (Graph Attention Network) learns attention weights for each neighbor, dynamically determining which neighbors are more important for the task. Use GCN when: the graph has uniform neighbor importance (citation networks where all cited papers are equally relevant), interpretability is not required, computational resources are limited (GAT is 2-3x more expensive per layer due to attention computation), or the graph is relatively small and simple. Use GAT when: neighbor importance varies significantly (fraud detection where some transactions are more suspicious than others, recommendation where some items are more influential), you want to understand which connections matter most (attention weights provide interpretability), or the graph has high-degree nodes where selective attention is beneficial. In practice, GAT generally outperforms GCN on most benchmarks (2-5% improvement), but GCN trains faster and uses less memory. For production systems, GAT is preferred when the budget allows, especially on heterophilic graphs where connected nodes are dissimilar. + +**Q: What is message passing in GNNs?** + +A: Message passing is the fundamental computation framework for GNNs. In each layer, every node in the graph: (1) Gathers messages from its neighboring nodes. A message is computed by transforming the neighbor's feature vector through a learned function (typically a linear transformation W * h_neighbor, optionally with edge features). (2) Aggregates all incoming messages using a permutation-invariant function — sum (most common, captures complete neighborhood information), mean (averages out noise, good for high-degree nodes), max (captures strongest signal, good for detecting outliers), or attention-weighted sum (GAT). (3) Updates its own representation by combining its previous representation (from the previous layer) with the aggregated neighbor messages — typically through concatenation followed by a non-linear transformation (ReLU). After k layers, each node's representation encodes information from its k-hop neighborhood. This locality principle is what makes GNNs powerful: they learn node representations that reflect the local graph structure. The parameters (weight matrices) are shared across all nodes, so the same update function applies everywhere — this is what allows GNNs to generalize to graphs of different sizes and structures. + +### Related Concepts +RAG, Knowledge Graphs, GNN, GraphRAG + +--- + +## Fine-tuning & Adaptation Methods +**Definition:** Techniques to adapt pre-trained models to specific tasks: full fine-tuning, parameter-efficient methods (LoRA, QLoRA), preference optimization (RLHF, DPO). + +### Theory & Explanation + +Fine-tuning is the process of adapting a pre-trained model to a specific task or domain by continuing training on task-specific data. The fundamental principle underlying all fine-tuning is transfer learning: a model pre-trained on internet-scale data has already learned general features of language (grammar, reasoning, factual knowledge), and fine-tuning only needs to adapt these general abilities to the specific target domain rather than learning from scratch. This makes fine-tuning dramatically more data-efficient than training from scratch — you can achieve strong performance with as few as 100-1000 high-quality examples, whereas pre-training requires trillions of tokens. However, fine-tuning comes with risks: catastrophic forgetting (the model loses its general capabilities while specializing), overfitting (the model memorizes the fine-tuning data rather than learning patterns), and distribution shift (the fine-tuning data distribution differs from real-world usage). + +Full fine-tuning updates all parameters of the model. This achieves the highest potential quality because every weight can adapt to the target task. However, it is computationally expensive — fine-tuning a 7B parameter model requires roughly 56GB of GPU memory for full precision (16-bit) training, and a 70B model requires 560GB+ (requiring 8x A100 80GB GPUs). Full fine-tuning also produces a complete copy of the model for each fine-tuned version, which is storage-intensive (13GB for a 7B model, 140GB for 70B). The key hyperparameters are learning rate (typically 1e-5 to 5e-5, much lower than pre-training), batch size, number of epochs (typically 1-3, more epochs increase overfitting risk), and warmup steps. Learning rate scheduling uses cosine decay or linear decay. Full fine-tuning is most appropriate when: you have large amounts of task-specific data (10K+ examples), you need maximum possible quality, you have the compute budget, and you are not fine-tuning for many different tasks (each requires storing a full copy). + +LoRA (Low-Rank Adaptation, Hu et al. 2021) is the most widely used parameter-efficient fine-tuning (PEFT) method. Instead of updating the full weight matrix W, LoRA injects two low-rank matrices A and B such that W' = W + BA, where A is (input_dim x r) and B is (r x output_dim), with r much smaller than the hidden dimension (typically r=8-64 vs d=4096 for a 7B model). The original weight W is frozen (not updated), and only A and B are trained. This reduces trainable parameters from 100% to roughly 0.1-0.5% — a 7B model fine-tuned with rank-8 LoRA trains only about 4 million parameters. The memory savings are substantial: optimizer states (Adam stores 2 states per parameter) are drastically reduced, enabling fine-tuning on consumer GPUs. LoRA adapters can be merged into the original weights at inference time (W + BA computed once), resulting in zero inference latency overhead and no increase in model size at serving time. Multiple LoRA adapters can be trained for different tasks and swapped at inference time without reloading the base model — enabling efficient multi-task serving from a single base model. LoRA is typically applied to attention projection matrices (Q, K, V, O), but can also be applied to FFN layers. + +QLoRA (Quantized LoRA, Dettmers et al. 2023) combines LoRA with 4-bit quantization of the base model. The base model weights are quantized to 4-bit NormalFloat (NF4) format, which uses a normalization trick to better represent the distribution of neural network weights. LoRA adapters are trained in full precision (16-bit or 32-bit) on top of the quantized base. During training, the quantized weights are dequantized on the fly to the computation precision for each forward pass, gradients flow through the LoRA adapters only, and the base weights remain frozen and quantized. QLoRA reduces memory by roughly 4x compared to full fine-tuning — a 7B model needs roughly 8GB (vs 56GB for full), a 13B model needs roughly 16GB (vs 104GB), and a 65B model can fit on a single 48GB GPU (vs 520GB). Performance: QLoRA matches LoRA quality within 0.5-1% on most benchmarks, and LoRA matches full fine-tuning within 1-2% when using sufficient rank (r >= 16). This means QLoRA achieves roughly 95-99% of full fine-tuning quality at 5-10% of the memory cost. QLoRA made fine-tuning accessible to consumer hardware and is the dominant approach for open-source model adaptation. + +DoRA (Weight-Decomposed Low-Rank Adaptation, Liu et al. 2024) improves upon LoRA by decomposing pre-trained weights into magnitude and direction components, then applying low-rank updates to the direction component. This is inspired by the observation that full fine-tuning primarily changes the direction of weight vectors rather than their magnitude. DoRA achieves better training stability and quality than LoRA at the same rank, and it converges faster (2-3x fewer steps). DoRA is implemented as a drop-in replacement for LoRA in most frameworks (HuggingFace PEFT). Adapters (Houlsby et al. 2019) are an earlier PEFT method that inserts small bottleneck layers (down-projection, ReLU, up-projection) into each Transformer layer. They are effective but add inference latency (the adapter is part of the forward pass and cannot be merged). Prefix Tuning (Li & Liang 2021) and Prompt Tuning (Lester et al. 2021) learn soft tokens (continuous embeddings) that are prepended to the input. These are even more parameter-efficient than LoRA (as few as 100-1000 learned tokens) but generally underperform LoRA on complex tasks. + +Preference optimization aligns models with human values and preferences after instruction tuning. RLHF (Reinforcement Learning from Human Feedback, used by OpenAI for ChatGPT/GPT-4) is a three-stage process: (1) Supervised fine-tuning (SFT) on high-quality demonstrations. (2) Train a reward model on human preference data — humans compare pairs of model outputs and choose the better one; the reward model learns to predict the human preference. (3) Optimize the policy (the LLM) using Proximal Policy Optimization (PPO), where the reward model provides the reward signal and a KL penalty prevents the policy from drifting too far from the SFT model. RLHF is complex and unstable — it requires training and maintaining a separate reward model, PPO is sensitive to hyperparameters, and the KL penalty introduces a balancing act between reward maximization and output diversity. + +DPO (Direct Preference Optimization, Rafailov et al. 2023) eliminates the reward model entirely. It directly optimizes the policy on preference pairs using a binary cross-entropy loss that implicitly represents the reward as a function of the policy's own likelihood ratio between preferred and dispreferred outputs. DPO is simpler, more stable, and requires less compute than RLHF (no reward model training, no PPO), and typically matches or outperforms RLHF in practice. The trade-off: DPO cannot easily incorporate non-pairwise feedback (ratings, multi-way comparisons) and cannot use explicit reward signals from external systems. ORPO (Odds Ratio Preference Optimization, Hong et al. 2024) combines SFT and preference optimization into a single stage by adding an odds ratio loss to the standard language modeling objective, further simplifying the pipeline. The model simultaneously learns to generate the target output (supervised) and to prefer it over rejected alternatives (preference). This eliminates the two-stage SFT-then-DPO pipeline, reducing training time by roughly 40%. + +### Example + +Fine-tuning LLaMA-2 7B on legal document summarization. Options and resource requirements: Full fine-tuning needs 56GB GPU memory (A100 80GB or 2x RTX 6000). LoRA (rank=8): 16GB memory (single RTX 3090/4090). QLoRA (4-bit NF4, rank=8): 8GB memory (single RTX 3080/4080). Training data: 5,000 legal case summaries from US federal courts. Training time: full - 12 hours on 4x A100; LoRA - 4 hours on single A100; QLoRA - 6 hours on single RTX 4090. Quality comparison on ROUGE-L: full fine-tune - 42.5; LoRA (r=8) - 41.8; LoRA (r=16) - 42.1; QLoRA (r=8) - 41.5; QLoRA (r=16) - 41.9. Conclusion: LoRA/QLoRA achieves 97-99% of full fine-tuning quality at 15-85% of the memory cost. The LoRA adapter checkpoint is 10MB vs 13GB for full fine-tuning. For production, use LoRA r=16 for the best quality-efficiency trade-off, or QLoRA for prototyping on consumer hardware. + +Preference optimization example: After SFT on legal summaries, the model still occasionally hallucinates court names and dates. A preference dataset of 2,000 pairs is created (each pair: a good summary and one with hallucination). DPO training for 1 epoch with beta=0.1. The trained model shows 60% fewer hallucinations while maintaining ROUGE scores. The KL divergence from the SFT model is monitored — if it exceeds 0.5, early stopping prevents forgetting. + +### Interview Questions + +**Q: LoRA vs Full Fine-tuning — tradeoffs?** + +A: Full fine-tuning updates all model parameters, offering the highest potential quality but requiring massive compute (56GB for 7B, 560GB+ for 70B), long training times, and producing large checkpoints (13GB for 7B). It risks catastrophic forgetting of general capabilities. LoRA updates only 0.1-0.5% of parameters via low-rank matrices injected into attention layers. Memory savings: roughly 3-4x less GPU memory (16GB vs 56GB for 7B). Training speed: 2-3x faster. Checkpoint size: ~10MB vs 13GB. Quality gap: LoRA typically achieves 97-99% of full fine-tuning quality at r >= 16. LoRA supports hot-swapping multiple adapters for different tasks without reloading the base model. Use full fine-tuning when: you need maximum quality, have large datasets (10K+), and have the compute budget. Use LoRA when: you need to fine-tune for multiple tasks, have limited compute, or are iterating rapidly on experiment design. In most production scenarios, LoRA with r=16-64 is the recommended starting point. + +**Q: RLHF vs DPO — differences?** + +A: RLHF (Reinforcement Learning from Human Feedback) is a three-stage process: (1) SFT on demonstrations. (2) Train a separate reward model on human preference comparisons (trained to predict which output a human would prefer). (3) Optimize the LLM using PPO against the reward model with a KL penalty. RLHF is complex, requires training two models (the reward model and the policy), PPO is hyperparameter-sensitive and unstable, and the reward model can be exploited (reward hacking). DPO (Direct Preference Optimization) eliminates the reward model: it directly optimizes the policy on preference pairs using a loss function that implicitly represents the reward as a ratio of policy probabilities. DPO is simpler (single training stage, no reward model), more stable (no PPO), and requires less compute (roughly 40% less total training time). DPO typically matches or slightly outperforms RLHF on standard benchmarks (AlpacaEval, MT-Bench). However, RLHF can incorporate additional reward signals (safety classifiers, task-specific metrics) that DPO cannot easily use. RLHF also has more research behind it for safety alignment. For most purposes, DPO is the recommended starting point due to its simplicity and stability. + +**Q: What is catastrophic forgetting and how to prevent it?** + +A: Catastrophic forgetting occurs when a model loses previously learned capabilities while being fine-tuned on a new task. For example, fine-tuning a general instruction-following model exclusively on legal summarization may cause it to lose translation, coding, or general QA abilities. This happens because weight updates for the new task overwrite weights that were important for old tasks. Prevention strategies: (1) Mix in general data — include 10-20% general domain examples during fine-tuning to maintain broad capabilities. (2) Elastic Weight Consolidation (EWC) — add a regularization term that penalizes changes to weights that were important for previous tasks (measured by the Fisher information matrix). (3) LoRA — by limiting the number of trainable parameters and freezing the base model, LoRA inherently reduces forgetting because the base model weights (containing general knowledge) remain untouched. (4) Multi-task fine-tuning — fine-tune on a mixture of tasks simultaneously rather than sequentially. (5) Replay buffers — periodically retrain on samples from previous tasks. (6) Early stopping — stop fine-tuning before forgetting sets in. (7) Low learning rates — use 1e-5 or lower to prevent large weight updates. In practice, LoRA with a diverse training mixture is the most effective and practical approach. + +### Related Concepts +LLMs, RLHF, LoRA, Pre-training + +--- + +## Evaluation & Hallucination in LLMs +**Definition:** Methods to assess LLM output quality and detect hallucinations: intrinsic metrics, LLM-as-judge, human evaluation, specialized RAG evaluation frameworks. + +### Theory & Explanation + +Evaluating LLM outputs is fundamentally challenging because the notion of correctness is often subjective and task-dependent. Unlike traditional machine learning where you can compute accuracy against a fixed label, LLM outputs are free-form text where multiple equally valid answers exist. This has led to a multi-layered evaluation ecosystem spanning automated metrics, LLM-based judges, human evaluation, and task-specific frameworks. The field is rapidly evolving because traditional NLG metrics developed for machine translation and summarization (BLEU, ROUGE, METEOR) are poorly suited for evaluating open-ended generative tasks. These metrics rely on n-gram overlap between the generated text and one or more reference texts, which cannot capture semantic equivalence. Two responses with identical meaning but different wording receive low scores — for example, "The patient has hypertension" vs "The patient's blood pressure is elevated" would score poorly on BLEU despite being semantically equivalent. + +Traditional NLG metrics serve specific niches. BLEU (Bilingual Evaluation Understudy) measures n-gram precision — the proportion of n-grams in the generated text that appear in the reference text, with a brevity penalty to discourage short outputs. BLEU is most appropriate for machine translation where reference translations are available and lexical correctness matters. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures n-gram recall — the proportion of n-grams in the reference text that appear in the generated text. ROUGE-L uses longest common subsequence to capture sentence-level structure. ROUGE is most appropriate for summarization where the goal is to cover the key information from the reference. METEOR extends these by incorporating synonym matching (using WordNet or paraphrase tables), stemming, and explicit word order penalties. METEOR correlates better with human judgment than BLEU or ROUGE alone. Perplexity is an intrinsic metric — the exponential of the negative log-likelihood of the text under the model. Lower perplexity means the model assigns higher probability to the text. Perplexity is useful for comparing models during pre-training but correlates poorly with task-specific quality. + +LLM-as-Judge has emerged as the dominant evaluation method for open-ended tasks. The approach: use a strong LLM (GPT-4, Claude 3.5, Gemini 1.5 Pro) as an evaluator, providing it with the question, the model's response, and evaluation criteria. The LLM judge rates the response on a scale or compares two responses (pairwise comparison). G-Eval (Liu et al. 2023) chains evaluator prompts through multiple steps: first define evaluation criteria, then ask the LLM to generate a chain-of-thought evaluation, then output a score 1-5. LLM-as-Judge correlates well with human judgment (0.5-0.7 Spearman correlation, approaching human inter-annotator agreement) but has biases: position bias (preferring the first or second response in pairwise comparison), verbosity bias (preferring longer responses), self-enhancement bias (preferring responses from the same model family), and format bias (preferring structured outputs). Mitigations include: shuffling response order, using calibrated scoring rubrics, having multiple judges and averaging, and using fine-tuned evaluator models (PandaLM, Prometheus). + +RAG evaluation requires specialized frameworks because RAG systems have two interacting components (retrieval and generation) and must be evaluated on both separately. The RAGAS (RAG Assessment) framework defines four core metrics computed without ground-truth answers (reference-free evaluation). (1) Context Precision: measures whether the retrieved chunks are relevant to the question. It uses the LLM to judge each retrieved chunk's relevance, then computes a precision score weighted by rank (relevant chunks appearing earlier get higher weight). (2) Context Recall: measures whether all necessary information to answer the question was retrieved. The LLM decomposes the ground-truth answer into claims and checks if each claim is supported by the retrieved context. (3) Faithfulness: measures whether the generated answer is consistent with the retrieved context. The LLM extracts claims from the generated answer and checks each against the context — claims not supported by context reduce faithfulness. (4) Answer Relevancy: measures whether the generated answer addresses the question. The LLM generates hypothetical questions from the answer and computes cosine similarity with the original question. RAGAS also provides synthetic test set generation — given a document corpus, it uses the LLM to generate questions and ground-truth answers covering different difficulty levels and question types. + +Hallucination detection in LLM outputs is a critical safety concern, especially in high-stakes domains (healthcare, legal, finance). Hallucinations fall into two categories: intrinsic hallucinations (the output contradicts the provided context or known facts) and extrinsic hallucinations (the output adds information not verifiable from any source). Detection methods include: (1) SelfCheckGPT — generate multiple responses to the same prompt with high temperature, then check for consistency across responses. If the model consistently mentions the same facts, they are likely grounded; if a fact appears in only some responses, it may be hallucinated. SelfCheckGPT uses NLI (natural language inference) models to check entailment between sentences from different samples. (2) NLI-based detection — train or use a pre-trained NLI model (e.g., TrueTeacher, DeBERTa) to check if the generated text entails the provided context. Contradiction or neutral judgments indicate hallucination. (3) Semantic entropy (Kuhn et al. 2023) — cluster semantically equivalent meanings across multiple generations, then compute entropy over the clusters. High semantic entropy indicates uncertainty and likely hallucination. This is more principled than lexical entropy because multiple paraphrases of the same meaning should not contribute to uncertainty. (4) Lookahead verification — the LLM is prompted to verify its own output by checking each claim against the context, similar to self-consistency but in a single pass. (5) Retrieval-based verification — extract claims from the generated answer and verify each by retrieving supporting evidence from a trusted knowledge base. LangSmith and LangFuse provide tracing and evaluation platforms that instrument every step of the RAG pipeline — logging retrieved chunks, generation parameters, scores, and latency — enabling debugging of specific failure cases and monitoring of system drift over time. + +Human evaluation remains the gold standard for output quality, especially for nuanced tasks like creative writing, tone appropriateness, and domain-specific accuracy. Common human evaluation protocols include: (1) Likert scale ratings (1-5) on specific criteria (relevance, fluency, completeness, safety). (2) Pairwise comparisons (A vs B) which are easier for humans to judge reliably than absolute scores. (3) Best-worst scaling — annotators select the best and worst from three or four options, producing more consistent rankings than pairwise. (4) Task-specific evaluation — for summarization, human judges check factuality, coverage, and conciseness. Human evaluation is expensive and slow (a typical study costs $5,000-50,000) but captures aspects no automated metric can. The practical approach for production systems is: use automated metrics and LLM-as-Judge for rapid iteration during development, use RAGAS for RAG system monitoring, and run periodic human evaluation studies to calibrate automated metrics and detect systematic issues like bias, toxicity, and subtle hallucination that automated methods miss. + +### Example + +RAG evaluation in practice: A customer support RAG system has a knowledge base of 1,000 product documentation pages. The evaluation dataset contains 200 queries collected from real customer support tickets, each with a ground-truth answer written by a human agent. For a sample query "What is your return policy for opened electronics?" the system retrieves chunk: "Our return policy allows returns within 30 days of purchase for unopened items. Opened electronics may be subject to a 15% restocking fee." The LLM generates: "You can return opened electronics within 30 days, but a 15% restocking fee applies." RAGAS evaluation: faithfulness = 0.95 (the answer is consistent with the context), answer_relevancy = 0.88 (the answer addresses the query), context_precision = 0.72 (one irrelevant chunk was also retrieved — about furniture returns), context_recall = 0.85 (the ground-truth answer mentions "receipt required" which the retrieved context did not cover). Compare with a hallucinated response: "You can return opened electronics for a full refund within 90 days." Faithfulness = 0.12 — detected immediately. The system also uses SelfCheckGPT: generating 5 responses with temperature 0.7, an NLI model checks consistency. The hallucinated claim "90-day return" appears in only 1 of 5 responses, triggering an alarm. The production monitoring dashboard shows a weekly faithfulness score trend: if it drops below 0.90, the team investigates by examining recent knowledge base updates, embedding model changes, or LLM provider updates. + +### Interview Questions + +**Q: How to evaluate RAG system output quality?** + +A: RAG evaluation must assess both retrieval and generation quality independently. For retrieval: hit rate (does the top-k contain at least one relevant chunk), MRR (rank of first relevant result), nDCG (graded relevance scoring), and precision at k. For generation: RAGAS metrics — faithfulness (is the answer grounded in retrieved context), answer relevancy (does it address the question), context precision (are retrieved chunks relevant), and context recall (is all necessary information retrieved). Additionally, use LLM-as-Judge with GPT-4 scoring outputs on criteria like helpfulness, accuracy, and safety. Pairwise comparisons (A vs B) with LLM judges provide reliable rankings. For production monitoring, track: end-to-end latency (ideally <2s), user satisfaction (thumbs up/down), citation accuracy (are cited sources real), and fallback rate (how often does the system say it doesn't know). Periodically run human evaluation studies to calibrate automated metrics. The evaluation set should include edge cases: ambiguous queries, multi-part questions, out-of-scope requests, and previously unseen topics. + +**Q: Methods to detect hallucinations in LLM outputs?** + +A: Hallucination detection methods fall into several categories. (1) Consistency-based: SelfCheckGPT generates multiple responses and checks for contradictions between them using an NLI model. If the model consistently produces the same fact across temperature variations, it is likely grounded. (2) Entropy-based: semantic entropy clusters responses by meaning and computes entropy across clusters — high entropy (many different semantic clusters) indicates uncertainty. (3) Verification-based: extract atomic claims from the generated text and verify each against a trusted knowledge base or the provided context. Each claim is classified as supported, contradicted, or unverifiable. (4) Probability-based: low token probabilities, high perplexity, and high predictive entropy correlate with hallucination risk. (5) LLM-as-judge: ask a strong LLM evaluator to check the response for factual accuracy against the context, using chain-of-thought prompting for detailed verification. In production, a combination is recommended: self-consistency as a real-time signal (high computational cost but accurate), NLI-based verification for post-hoc monitoring, and adversarial evaluation during testing (deliberately provide contexts with contradictions and check if the model follows context or hallucinates). + +**Q: BLEU vs ROUGE vs METEOR — differences and when to use?** + +A: BLEU (Bilingual Evaluation Understudy) measures n-gram precision — the percentage of n-grams in the generated text that appear in the reference. It includes a brevity penalty to penalize short outputs. BLEU is best for machine translation where multiple valid translations share significant vocabulary. ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures n-gram recall — the percentage of reference n-grams that appear in the generated text. ROUGE-L uses the longest common subsequence (LCS) to measure sentence-level structure similarity. ROUGE is best for summarization where the goal is to cover all key information from the reference. METEOR extends both by: incorporating synonym matching (using WordNet, paraphrase tables), using stemming to match morphological variants, and adding a word order penalty. METEOR correlates better with human judgment than BLEU or ROUGE alone. Key differences: BLEU is precision-focused and penalizes extra words; ROUGE is recall-focused and penalizes missing words; METEOR balances both with linguistic knowledge. None of these capture semantic equivalence — two responses with identical meaning but different vocabulary will score poorly. For modern LLM evaluation, these metrics are largely superseded by embedding-based similarity (BERTScore, BLEURT) and LLM-as-Judge, which better capture semantic quality. + +### Related Concepts +RAG, LLMs, Evaluation Metrics, RAGAS + +--- + +## Additional LLM Concepts + +**1. Foundation Model** +**Definition:** Large pre-trained model adaptable to many downstream tasks without task-specific architecture changes. +### Theory & Explanation +Foundation models are trained on broad data at massive scale (internet text, code, images, audio) via self-supervised learning. Unlike traditional task-specific models (e.g., a BERT fine-tuned only for sentiment analysis), foundation models serve as a general-purpose base. The key insight is scaling — as model size, data, and compute grow, emergent abilities appear: in-context learning, reasoning, instruction following. This paradigm shift from "train a model for every task" to "pre-train once, adapt for many tasks" has reshaped NLP. +### Example +GPT-4 can serve as the foundation for a chatbot, a code assistant (GitHub Copilot), a data analyst (code interpreter), or a creative writing tool — all without architectural changes, only different prompts and possibly fine-tuning. Claude and LLaMA follow the same paradigm. +### Interview Questions +**Q: How do foundation models differ from traditional task-specific models?** +A: Foundation models are trained once on diverse data at scale and adapted via prompting or fine-tuning for many tasks. Task-specific models require separate training and architecture for each task (e.g., one BERT for QA, another for NER). Foundation models exhibit emergent abilities that task-specific models lack. + +**Q: What scaling laws govern foundation model performance?** +A: Kaplan et al. (2020) showed that model performance follows a power-law relationship with model size, dataset size, and compute. Chinchilla scaling laws (Hoffmann et al., 2022) refined this, showing most models are undertrained — for optimal performance, model parameters and training tokens should scale roughly equally (i.e., a 70B model should be trained on ~2T+ tokens). + +--- + +**2. Instruction Tuning** +**Definition:** Fine-tuning a pre-trained LLM on (instruction, response) pairs to improve its ability to follow directions and perform diverse tasks. +### Theory & Explanation +Base pre-trained models are next-token predictors — they complete text but don't naturally follow instructions. Instruction tuning bridges this gap by training on supervised examples where each input is a natural language instruction and each output is the desired response. The dataset can range from 1K to 50K high-quality examples. FLAN (Fine-tuned LAnguage Net) showed that instruction tuning on a mixture of tasks (many NLP datasets reformatted as instructions) enables zero-shot generalization to unseen tasks. InstructGPT used RLHF after instruction tuning to align with human preferences. Key insight: instruction tuning teaches format and intent understanding, not task-specific knowledge. +### Example +A base GPT-3 might respond to "Summarize this article" by continuing the article text. An instruction-tuned version produces a concise summary. Example instruction data point: {"instruction": "Classify this sentiment", "input": "I loved the movie!", "output": "Positive"}. +### Interview Questions +**Q: How does instruction tuning differ from RLHF?** +A: Instruction tuning is supervised fine-tuning on (instruction, response) pairs — the model learns to imitate correct responses. RLHF (Reinforcement Learning from Human Feedback) goes further: train a reward model on human preferences, then optimize the LLM to maximize reward. Instruction tuning teaches format; RLHF teaches values and preferences. They are often used together: instruction tuning first, then RLHF. + +**Q: Why is data quality more important than quantity for instruction tuning?** +A: Low-quality examples (wrong, vague, or hallucinated responses) teach the model bad patterns. A small set of diverse, high-quality, human-verified examples (e.g., 5K-10K) consistently outperforms large noisy datasets (100K+). The LIMA paper demonstrated this: training on just 1K carefully curated examples produced results competitive with 50K+ examples. + +--- + +**3. Logit Bias** +**Definition:** Adding a bias value to the logits of specific tokens before sampling, increasing or decreasing their probability during generation. +### Theory & Explanation +During auto-regressive generation, the model outputs logits (un-normalized scores) for all tokens in the vocabulary. Softmax converts these to probabilities. Logit bias adds a constant value to specific token logits before softmax: positive bias increases the token's probability, negative bias decreases it. The bias range is typically -100 to +100. A bias of +100 nearly guarantees the token appears; -100 nearly eliminates it. Logit bias is applied per-token at each generation step, unlike temperature which scales all logits uniformly. Most LLM APIs expose this parameter (OpenAI: `logit_bias`, Anthropic: `logit_bias`). +### Example +To force JSON output, apply +100 bias to tokens like `{`, `}`, `"`, `:`, `,` and -100 bias to natural language tokens. To reduce harmful output, apply a small negative bias (-5 to -10) to tokens associated with violence or profanity. In OpenAI API: `{"logit_bias": {"42": -100, "": 10}}` maps token IDs to bias. +### Interview Questions +**Q: What's the difference between logit bias and temperature?** +A: Temperature scales all logits uniformly (dividing by T), flattening or sharpening the entire probability distribution. Logit bias targets specific token IDs individually. Temperature affects randomness; logit bias affects content constraints. They can be combined: use temperature for creativity, logit bias for format enforcement. + +**Q: When would you use logit bias vs constrained decoding?** +A: Logit bias is simpler and works with any model API that exposes it but doesn't guarantee constraint satisfaction (a biased token can still be outvoted by many other tokens). Constrained decoding (e.g., guidance, LMQL, Outlines) enforces hard constraints using grammars or schemas, guaranteeing output structure. Use logit bias for soft preferences (discourage certain words); use constrained decoding for hard requirements (valid JSON, SQL, code with correct syntax). + +--- + +**4. Beam Search** +**Definition:** A decoding strategy that maintains the top-b candidate sequences at each step, selecting the one with the highest overall probability at the end. +### Theory & Explanation +Greedy decoding chooses the highest-probability token at each step, which can miss better sequences — a locally suboptimal choice early on can lead to a globally worse sequence. Beam search keeps b candidates (the "beam"), expanding each by considering all possible next tokens, then retaining the top b candidates by cumulative log probability. At termination, the highest-scoring complete sequence is selected. Beam width b=2-10 is typical. Larger b improves quality but increases computation (O(b × vocabulary × sequence_length)). However, wider beams can reduce diversity and introduce repetition. Beam search is deterministic (same input → same output), making it suitable for tasks where quality and consistency matter over creativity. +### Example +For machine translation of "Je suis étudiant" to English with b=2: Step 1: ["I" (prob=0.6), "I'm" (prob=0.3)]. Step 2: expand both — ["I am" (0.42), "I'm a" (0.12), "I'm an" (0.09), "I have" (0.06)]. Keep top 2: ["I am", "I'm a"]. Step 3: continue until EOS. Final: "I am a student" beats greedy "I am student" because beam search saw "I am a" had higher joint probability. +### Interview Questions +**Q: What are the trade-offs of increasing beam width?** +A: Wider beams find higher probability sequences (better quality) but at quadratic computational cost. They also reduce output diversity (all b sequences converge to similar outputs) and can introduce repetition (the model favors high-frequency patterns). For creative tasks (storytelling, dialogue), beam width 1 (greedy) or sampling is preferred. For factual tasks (translation, summarization), beam width 3-5 is typical. + +**Q: How does beam search compare with greedy decoding and sampling?** +A: Greedy is fastest but can miss optimal sequences. Beam search is slower but higher quality and deterministic. Sampling introduces randomness for diversity but can produce incoherent output. Hybrid approaches exist: beam search with temperature (sample within the beam), or diverse beam search (add diversity penalty to beam candidates). + +--- + +**5. Decoding Strategies** +**Definition:** Algorithms for selecting tokens during auto-regressive generation, balancing quality, diversity, and speed. +### Theory & Explanation +All decoding strategies solve the same problem: given model probabilities for the next token, how to choose it. The spectrum ranges from deterministic to fully random. Greedy (argmax at each step): fast, low quality, repetitive. Beam search: best quality for factual tasks, slow. Top-k sampling: sample from the k most likely tokens (k=40-100), controlled randomness. Top-p (nucleus) sampling: sample from the smallest set of tokens whose cumulative probability exceeds p (p=0.9-0.95), adaptive — more tokens when model is uncertain, fewer when confident. Temperature: scale logits by 1/T before softmax. T → 0 approaches greedy, T → ∞ approaches uniform sampling. T < 1 sharpens distribution (more conservative), T > 1 flattens it (more creative). In practice, these are combined: temperature + top-p + top-k together control randomness at different levels. +### Example +A model predicting the next token after "The capital of France is": greedy picks "Paris" (prob=0.7). Top-k with k=3 samples from ["Paris" (0.7), "London" (0.1), "Berlin" (0.05)]. Top-p with p=0.9 includes tokens until cumulative prob ≥ 0.9: ["Paris" (0.7), "London" (0.1), "Berlin" (0.05), "Madrid" (0.03), "Rome" (0.02)]. Temperature T=2.0 flattens all probabilities, making unlikely tokens more plausible. +### Interview Questions +**Q: When should you use each decoding strategy?** +A: Greedy: baseline, speed-critical apps, deterministic needs. Beam search: factual generation (translation, summarization) where quality > speed. Top-k/top-p: creative generation (storytelling, dialogue, brainstorming) where diversity matters. Temperature: fine-grained creativity control — low (0.1-0.3) for code generation (correctness), medium (0.7-0.9) for creative writing, high (1.2-1.5) for idea generation. + +**Q: Why is combining top-k and top-p recommended over using either alone?** +A: Top-k has a fixed cutoff — if k is too small, it truncates valid tokens when the distribution is flat; if too large, it includes unlikely tokens when the distribution is peaked. Top-p adapts to distribution shape but can include many low-probability tokens in flat distributions. Together: first filter to top-k tokens, then apply top-p on the filtered set — this gives both a hard cap on candidates and adaptive thresholding. + +--- + +**6. Notable LLMs: Mistral, Falcon, Qwen** +**Definition:** Prominent open-weight LLM families with distinct architectural innovations and trade-offs. +### Theory & Explanation +Mistral AI released Mistral-7B (2023) which matched or outperformed LLaMA 2 13B using sliding window attention (each token attends to its local window of 4096 tokens, saving compute while maintaining long-range performance through stacked layers). The Mixtral 8x7B model uses mixture-of-experts (MoE) — 8 expert sub-networks with 2 active per token, achieving 46.7B total params but only 12.9B active, rivaling Llama 2 70B at fraction of the compute. Falcon (TII, UAE): Falcon-40B and 180B use multi-query attention (all heads share key/value projections, saving memory). Falcon-180B rivals PaLM-2 on benchmarks. Qwen (Alibaba): Qwen (Qianwen) series offers 128K+ native context length, strong multilingual performance (Chinese + English), and tool-use abilities. Qwen 2.5 72B competes with LLaMA 3 70B on many benchmarks. +### Example +Mistral-7B outperforms LLaMA 2 13B on most benchmarks while being 45% smaller. Falcon-180B achieves 68.5% on MMLU (near GPT-3.5 levels) while being fully open-weight. Qwen 2.5 72B scores 85+ on MMLU-Pro and handles 128K context natively. +### Interview Questions +**Q: What architectural innovations did Mistral introduce?** +A: Mistral's key innovation is sliding window attention — each layer's attention is limited to a fixed window (4096 tokens), but information propagates through layers (a token at position 1 in layer 1 can attend to position 4097 in layer 2). This gives O(L × w) instead of O(L²) attention cost. Mistral also uses MoE (Mixtral 8x7B) where different experts handle different token types, achieving better compute-to-quality ratio. + +**Q: How do these models compare with LLaMA for production use?** +A: LLaMA has the strongest ecosystem (tools, quantization, community). Mistral offers the best performance-per-parameter ratio (especially mistral-small, mistral-medium APIs). Falcon is most open (true open weights, no restrictions for Falcon-180B). Qwen is best for multilingual scenarios (especially Chinese) and long-context applications (native 128K+). Choose based on: ecosystem need → LLaMA, cost efficiency → Mistral, regulatory compliance → Falcon, multilingual/long-context → Qwen. + +--- + +## Additional RAG Concepts + +**7. Indexing** +**Definition:** The pre-processing pipeline that converts raw documents into a searchable structure for retrieval. +### Theory & Explanation +Indexing in RAG involves: (1) Cleaning: remove boilerplate, HTML tags, irrelevant content, PII. (2) Chunking: split documents into manageable pieces (256-1024 tokens) with overlap (10-20%) to maintain context at boundaries. (3) Embedding: convert each chunk to a dense vector using an embedding model (text-embedding-3-small, BGE, E5). (4) Storing: store vectors in a vector database (Pinecone, Chroma, Weaviate, Qdrant) with an index structure (HNSW, IVF) for approximate nearest neighbor search. Most production RAG systems maintain two indexes: a vector index for semantic search and an inverted index (BM25) for keyword/lexical search, enabling hybrid retrieval. Index maintenance involves deciding between incremental updates (add new documents without full reindex) and full reindexing (rebuild entire index periodically, e.g., every 24 hours for news domains). +### Example +For a legal document: (1) Parse PDF, remove headers/footers/citations. (2) Chunk into 512-token segments with 50-token overlap. (3) Embed using text-embedding-3-small (1536 dimensions). (4) Store in Pinecone with HNSW index (efConstruction=128, M=16). (5) Also build BM25 index for hybrid retrieval — at query time, search both and fuse results. +### Interview Questions +**Q: What are the trade-offs between chunk sizes?** +A: Small chunks (128-256 tokens): more precise retrieval, relevant chunks are highly focused but may miss broader context and require more context window management at generation time. Large chunks (512-1024 tokens): richer context, better for complex reasoning but may include irrelevant content (diluting embedding quality) and exceed context windows. Optimal chunk size depends on content type — code benefits from larger chunks (function-level), while FAQ/QA benefits from smaller chunks (single Q&A). Overlapping chunks mitigates boundary issues. + +**Q: How do you handle index updates for a production RAG system?** +A: Three strategies: (1) Incremental: add/update individual document embeddings without rebuilding — efficient but index quality degrades over time (HNSW graph becomes suboptimal). (2) Periodic full reindex: rebuild from scratch every N hours/days — maintains quality but is compute-heavy and causes downtime during reindex. (3) Shadow index: build new index in parallel, swap on completion — zero downtime but requires 2x storage. Most production systems use incremental for real-time ingestion + shadow reindex on a schedule. + +--- + +**8. Similarity Search** +**Definition:** Finding the most similar vectors to a query vector using a distance or similarity metric. +### Theory & Explanation +The core operation in vector search: given a query embedding q and a collection of document embeddings D, find the top-k most similar. Three common metrics: (1) Cosine similarity: measures angle between vectors, range [-1, 1], invariant to vector magnitude. Formula: cos(q, d) = (q·d) / (||q|| ||d||). Most embedding models use cosine similarity. (2) Dot product: measures both angle and magnitude, q·d. When vectors are normalized (unit length), dot product = cosine similarity. (3) Euclidean (L2) distance: straight-line distance, sensitive to magnitude. For normalized embeddings, ranking by Euclidean distance is equivalent to ranking by cosine similarity (monotonic transformation). Pre-normalized embeddings are recommended for consistent behavior. Approximate Nearest Neighbor (ANN) algorithms (HNSW, IVF) trade 1-5% recall for 10-100x speed vs exact search. +### Example +Query: "How to train a neural network?" Embed to q (1536-dim vector). Compare against 1M document vectors. Cosine similarity scores: doc_A (chunk about backpropagation): 0.92, doc_B (chunk about gradient descent): 0.87, doc_C (chunk about Python lists): 0.12. Return top-3 docs A, B, C. Using HNSW (ef_search=256) finds same results in ~10ms vs ~500ms for exact search. +### Interview Questions +**Q: Which similarity metric should you use with different embedding models?** +A: Follow the embedding model's recommendation: OpenAI text-embedding-3 models are trained for cosine similarity. BGE models can use either cosine or dot product (they provide instructions). Cohere embed models prefer dot product. For normalized embeddings (most modern models), cosine, dot product, and Euclidean give equivalent rankings — the choice is implementation convenience. For unnormalized embeddings (rare), dot product captures both similarity and magnitude which may conflate relevance with verbosity. + +**Q: What's the difference between exact search and ANN (approximate nearest neighbor)?** +A: Exact search (brute force) computes distance against every vector — guaranteed correct but O(N) per query, impractical at scale (>100K vectors). ANN uses index structures: HNSW (hierarchical navigable small world graphs, best recall/speed), IVF (inverted file index, good for very large collections), or product quantization (compression for memory efficiency). ANN trades perfect recall for speed — typical recall@10 is 95-99% with 10-100x speedup. Use ANN for latency-sensitive apps, exact search for small collections or where perfect recall is critical. + +--- + +**9. Query Expansion** +**Definition:** Generating multiple variants of a user's query and searching all of them to improve retrieval recall. +### Theory & Explanation +A single user query may not contain the exact terms used in relevant documents. Query expansion addresses this by generating multiple reformulations: (1) Synonym expansion: replace key terms with synonyms (WordNet, thesaurus). (2) Back-translation: translate query to another language and back to generate paraphrases. (3) LLM-generated variations: ask an LLM to produce 3-5 alternative phrasings of the query. (4) Query decomposition: break complex queries into sub-queries. Each variant is searched independently, and results are merged (deduplicated, re-ranked). This increases recall (finding more relevant documents) at the cost of latency (multiples of single query time) and potential recall degradation (noise from poor expansions). Expansion is most effective for short, ambiguous queries and domain-specific terminology. +### Example +User query: "How do I fix memory leaks in Python?" → Expand to: (1) "Resolve memory leaks Python", (2) "Python memory management best practices", (3) "Garbage collection issues Python", (4) "Python memory leak debugging tools". Search all 4, collect unique results, rerank by relevance to original query. This finds documents about gc, weakref, and tracemalloc that the original query might miss. +### Interview Questions +**Q: When is query expansion not beneficial?** +A: Query expansion hurts when: (1) the original query is already precise and well-formed (long, specific queries don't benefit much). (2) The expansion introduces noise — poor paraphrases retrieve irrelevant documents. (3) Latency is critical — 5 queries vs 1 increases latency 3-5x even with parallel execution. (4) The domain has standardized terminology where synonyms are misleading (e.g., "Python" the language should not expand to "snake"). Use expansion selectively based on query characteristics. + +**Q: Compare query expansion vs query transformation.** +A: Query expansion generates multiple search variants then merges results. Query transformation modifies the query before search (e.g., rewriting a question into a declarative statement, translating to English). Expansion increases breadth (more terms); transformation increases precision (better alignment with document structure). They are complementary: transform first to create a well-formed query, then expand for recall. + +--- + +**10. Context Window Management** +**Definition:** Strategies for selecting and ordering retrieved chunks to fit within the LLM's context window while maximizing relevance and coherence. +### Theory & Explanation +LLMs have a fixed context window (4K-200K tokens depending on model). In RAG, retrieved chunks often exceed this limit. Context window management decides: (1) Which chunks to include (filtering, relevance thresholding). (2) How many chunks to include (budget allocation). (3) How to order them (relevance descending, original document order, or structured template). (4) What to do when chunks exceed the window. Strategies include: relevance-weighted truncation (drop lowest relevance chunks first), sliding window (include adjacent chunks for each retrieved chunk), summarization of older or less relevant context, and structured formatting ("Context:\n{doc1}\n\n{doc2}\n\nQuestion: {q}"). A common heuristic is: put the most relevant chunks first (LLMs pay more attention to early tokens), reserve 10-20% of the window for the instruction and question, and keep total context under 70% of the model's max window for safety margin. +### Example +Given: 15 retrieved chunks (avg 200 tokens each = 3000 tokens), LLM window 4096 tokens, instruction + question = 500 tokens. Budget: 4096 - 500 = 3596 remaining. Sort chunks by relevance score, take top 15 (3000 tokens, fits). If chunks were larger (500 each → 7500 tokens), take top 7 (3500 tokens), drop rest, or summarize additional chunks and append as compressed context. +### Interview Questions +**Q: What are the trade-offs of stuffing all retrieved chunks vs selective inclusion?** +A: Stuffing all chunks provides maximum information but risks: (1) exceeding context window (hard truncation of tail chunks). (2) Diluting attention — LLMs attend less to information in the middle (lost-in-the-middle phenomenon). (3) Increased latency and cost (more tokens processed). Selective inclusion improves focus and latency but may miss relevant information. Optimal approach: relevance-thresholded selection (discard docs below similarity 0.7) + top-k limit (max 10 chunks) + reorder by relevance. + +**Q: How does the "lost in the middle" phenomenon affect RAG?** +A: Research (Liu et al., 2023) shows LLMs attend disproportionately to the beginning and end of long contexts, with information in the middle being "lost." For RAG, this means: place the most relevant chunks first, use separator tokens to demarcate chunks clearly, and if using many chunks, prioritize front-loaded ordering. Some systems use "recency bias" ordering (most relevant last) to exploit both the primacy effect (first) and recency effect (last). + +--- + +**11. Sliding Window Retriever** +**Definition:** Retrieves a chunk, then includes adjacent chunks (before and after) to provide coherent context, maintaining narrative flow. +### Theory & Explanation +Standard chunking breaks documents into fixed-size pieces, but the optimal information might be split across chunk boundaries. A sliding window retriever addresses this: when chunk i is retrieved (based on similarity to the query), also include chunks i-w to i+w (where w is the window size, typically 2-4 on each side). This preserves the surrounding context — the sentences before (providing introduction/setup) and after (providing conclusion/elaboration). The window helps when: (1) The answer spans multiple chunks. (2) Key context is in a neighboring chunk. (3) The retrieved chunk starts or ends mid-sentence. The trade-off is increased token usage (window of 3 on each side adds 6 chunks) and potential inclusion of irrelevant content. The window can be static (always include neighbors) or dynamic (include neighbors only if they improve coherence, measured by embedding similarity or entity overlap). +### Example +Document chunked into 128-token chunks. Chunk 7 retrieved (relevance 0.91). With window size 2: include chunks 5, 6, 7, 8, 9. Chunk 5 contains the topic introduction ("Memory management involves..."), chunk 6 continues, chunk 7 (the retrieved one) is the core explanation, chunks 8-9 contain code examples. Without the window, the LLM sees only the core explanation without context or examples. +### Interview Questions +**Q: When would you use a large window vs small window?** +A: Large window (5-10 on each side): narrative/creative content (stories, articles, long explanations) where context is diffuse and critical for coherence. Small window (1-2): factual/QA content where answers are localized (encyclopedia entries, code snippets, API docs). No window (0): each chunk is self-contained (FAQ, standalone definitions). The optimal window is determined empirically — evaluate answer completeness with different window sizes on a validation set. + +**Q: How does sliding window retrieval differ from overlap during chunking?** +A: Chunk overlap (typically 10-20%) ensures sentences aren't split mid-thought — adjacent chunks share content at boundaries. Sliding window retrieval includes complete neighboring chunks — it provides full surrounding context, not just boundary smoothing. These are complementary: use overlap during chunking for clean chunk boundaries, and sliding windows during retrieval for broader context when needed. + +--- + +**12. Contextual Compression** +**Definition:** Compressing or trimming retrieved documents to extract only the parts relevant to the query, reducing token usage and removing irrelevant content. +### Theory & Explanation +Not all content in a retrieved chunk is relevant to the query — a chunk about "Python memory management" might contain a sentence about history of Python that is irrelevant to "How to fix memory leaks?" Contextual compression solves this by passing each retrieved document through a compression step: (1) LLMChainExtractor: uses an LLM to extract only query-relevant sentences from each document. (2) LLMChainFilter: returns the entire document or nothing (binary relevance filter). (3) Embedding-based filtering: recompute similarity between query and each sentence within the document, keep only high-scoring sentences. Compression can reduce token usage by 40-70% while maintaining or improving answer quality (by removing distracting content). The trade-off is additional latency (compression step adds LLM call per document) and potential loss of useful context (aggressive compression may remove indirectly relevant content). +### Example +Retrieved chunk: "Python was created by Guido van Rossum in 1991. Memory management in Python uses reference counting and a generational garbage collector. The garbage collector handles circular references by periodically running cycle detection. Python's design philosophy emphasizes readability." Query: "How does Python's garbage collector work?" → Compressed: "Memory management in Python uses reference counting and a generational garbage collector. The garbage collector handles circular references by periodically running cycle detection." +### Interview Questions +**Q: What are the latency/cost implications of contextual compression?** +A: Each compressed document requires an LLM call (or at least embedding computation), adding O(k × d) latency where k = number of documents and d = LLM latency per call. For 10 documents at ~500ms each, that's 5 seconds added. Mitigations: (1) Use a smaller/faster LLM for extraction (e.g., GPT-4o-mini vs GPT-4o). (2) Filter first (top-3 documents only) then compress. (3) Use embedding-based compression (faster, no LLM call). (4) Cache compressed versions. In practice, compression is most valuable when document quality varies (mixed relevance within chunks) and latency is not the primary constraint. + +**Q: Compare LLMChainExtractor vs LLMChainFilter.** +A: LLMChainExtractor asks the LLM "extract relevant parts from this document given this query" — it produces a shorter, focused document. LLMChainFilter asks "is this document relevant?" — yes → keep entire document, no → discard. Extractor uses fewer tokens per document in generation (good for context window limits) but requires more LLM reasoning. Filter is faster (binary decision) but either includes or excludes entire documents. Extractor is better when documents are long with mixed relevance; filter is better when documents are short but many are irrelevant. + +--- + +**13. Fusion Retrieval** +**Definition:** Retrieving from multiple heterogeneous sources (vector DB, web search, SQL, APIs) and fusing results into a unified, deduplicated, re-ranked list. +### Theory & Explanation +No single retrieval source is optimal for all queries. Vector search excels at semantic similarity but may miss exact matches or recent information. Web search provides current information but can be noisy. SQL queries provide structured, exact data for entities and relationships. API calls can access domain-specific knowledge bases. Fusion retrieval dispatches the query to all sources in parallel, then: (1) Merge: collect results from all sources into a single pool. (2) Normalize: convert scores from different sources to a common scale (min-max normalization, rank-based recoding). (3) Deduplicate: remove near-duplicate chunks (by content hash or embedding similarity >0.95). (4) Re-rank: score each result by relevance to the original query using a cross-encoder or LLM. Common fusion algorithms: Reciprocal Rank Fusion (RRF) — each document scores sum(1/(k + rank_per_source)), simple and effective — or weighted fusion where each source has a learned importance weight. +### Example +Query: "Latest React 19 release date and features" → Dispatch to: (1) Vector DB (internal docs, semantic match). (2) Web search via Brave/Tavily (latest news). (3) GitHub API (check React releases). (4) NPM API (version metadata). Results: DB returns React 18 docs (outdated), web returns "React 19 released March 2025", GitHub returns release notes for v19.0.0, NPM returns version 19.0.0. RRF fuses: GitHub doc rank 1 → score 0.33, web result rank 2 → 0.25, DB docs (low relevance) → rank 25. Final top results include release notes, announcement, and changelog. +### Interview Questions +**Q: How do you handle source reliability differences in fusion retrieval?** +A: Not all sources have equal reliability. Strategies: (1) Static source weighting — trusted sources (curated DB, official APIs) get higher fusion weights. (2) Source-specific score normalization — use different scaling per source based on historical reliability. (3) Post-retrieval verification — use an LLM to verify facts against multiple sources; if sources disagree, flag for human review. (4) Freshness weighting — recent sources get higher weight for time-sensitive queries. In practice, a tiered approach works: high-authority sources (documentation, official APIs) → mid (curated web) → low (unstructured web). + +**Q: Compare fusion retrieval with hybrid search.** +A: Hybrid search combines sparse (BM25) and dense (embedding) retrieval from the same collection — different representations of the same data. Fusion retrieval combines different data sources — different data altogether. Hybrid search is one type of fusion (two sources: keyword and semantic indexes of the same corpus). Fusion retrieval generalizes to any number and type of sources. Hybrid search solves vocabulary mismatch; fusion retrieval solves coverage gaps. + +--- + +**14. Modular RAG** +**Definition:** A composable architecture where RAG functionality is decomposed into independent, interchangeable modules (query transformation, routing, fusion, memory, prediction, task adaptation). +### Theory & Explanation +Traditional RAG follows a fixed pipeline: retrieve → augment → generate. Modular RAG recognizes that different tasks and domains require different RAG configurations. It decomposes the system into modules: Query Transformation (rewrite, expand, decompose query), Routing (direct query to appropriate retriever), Fusion (combine multiple retrieval results), Memory (maintain conversation history across turns), Predict (generate final answer with different strategies), and Task-Adaptive (optimize for specific tasks like summarization, QA, or code generation). Each module can be independently configured, replaced, or skipped. For example, a customer support RAG might use: query transformation → routing to FAQ DB or product docs → fusion of results → memory context → generation with citation. A code assistant RAG might skip query transformation and use direct code search → fusion with web results → generation with code formatting. Frameworks like LlamaIndex and LangChain support modular RAG natively. +### Example +A legal RAG system: (1) Query Transformation: rewrite legal question into search-friendly terms ("What are the damages for breach of contract?" → "breach of contract damages calculation methods"). (2) Routing: if query mentions a specific jurisdiction → route to jurisdiction-specific legal DB; otherwise → general legal corpus. (3) Retrieval: hybrid search (BM25 + embedding) on legal documents. (4) Fusion: combine with recent case law from web search. (5) Memory: maintain client-specific case context. (6) Predict: generate with citations to specific statutes. +### Interview Questions +**Q: What are the advantages of modular over monolithic RAG?** +A: Modular RAG offers: (1) Flexibility — swap or skip modules per task without rewriting the system. (2) Testability — each module can be independently validated. (3) Observability — clear logging at each module boundary makes debugging easier. (4) Reusability — modules can be shared across different applications. (5) Progressive enhancement — start with a simple pipeline and add modules as needed. The trade-off is increased system complexity and potential latency from module orchestration overhead. + +**Q: How do you decide which modules a given RAG system needs?** +A: Start with the minimum viable RAG: retrieve → generate. Add modules based on failure modes: missing information → query expansion. Wrong information → hybrid retrieval + fusion. Incoherent across turns → memory module. Off-topic results → routing module. Hallucinations → verification/post-processing module. Add one module at a time and measure recall, precision, and latency impact. Most production RAG systems use 3-5 modules. + +--- + +**15. FLARE (Forward-Looking Active Retrieval Augmented Generation)** +**Definition:** An active retrieval method where the model predicts upcoming tokens, uses them to formulate a retrieval query mid-generation, and retrieves relevant context when confidence is low. +### Theory & Explanation +Standard RAG retrieves once before generation — all context is fixed upfront. FLARE retrieves multiple times during generation. The process: (1) Generate a temporary next sentence or span. (2) If the model has low confidence (probability < threshold) on tokens containing factual claims (dates, names, numbers), treat these as a retrieval query. (3) Search the knowledge base with this predicted query. (4) Retrieve relevant context and regenerate the low-confidence tokens conditioned on the new context. (5) Continue generation, repeating the process. This active retrieval ensures the model has access to the right information at the right time, especially for multi-step reasoning or long-form generation where information needs change as the answer develops. FLARE improves factual accuracy by 10-20% over single-retrieval RAG on knowledge-intensive tasks but adds significant latency (multiple retrieval steps per generation). +### Example +Generating: "The Eiffel Tower was completed in [low confidence → predict '1889', retrieve 'Eiffel Tower completion date 1889'] 1889. It was built for the [predict 'World's Fair', retrieve '1889 Exposition Universelle'] World's Fair. The tower is [predict '330', retrieve 'Eiffel Tower height 330m'] 330 meters tall." Each prediction triggers a targeted retrieval for that specific fact. +### Interview Questions +**Q: When is FLARE preferable to standard RAG?** +A: FLARE excels when: (1) Answers require multiple factual claims from different parts of the knowledge base (e.g., a biography: birth date, education, career milestones). (2) The information needed changes as generation progresses (you can't predict all needs upfront). (3) The query is ambiguous and retrieval benefits from seeing partial answers first. Standard RAG is preferable when: (1) The answer is contained in a single document. (2) Latency is critical (FLARE is 3-5x slower). (3) The knowledge base is small and well-indexed. + +**Q: How does FLARE handle low-confidence detection?** +A: FLARE monitors token-level probabilities during generation. It flags tokens where probability falls below a threshold (e.g., p < 0.5). But not all low-probability tokens need retrieval — the model focuses on "information tokens": named entities, dates, numbers, technical terms. It uses a lightweight classifier or regex to identify these token types. The flagged tokens form a retrieval query (e.g., extract entities from the low-confidence span and query the knowledge base). The threshold and entity detection rules are tuned per domain. + +--- + +**16. RAG vs Alternatives Comparison Table** +**Definition:** Structured comparison of RAG, fine-tuning, and prompt engineering across multiple dimensions. +### Theory & Explanation +Each approach has distinct strengths: RAG excels at incorporating new/changing information and providing citations. Fine-tuning excels at customizing behavior, tone, and domain-specific patterns. Prompt engineering is fastest and cheapest for simple tasks but limited in complexity. The choice is not mutually exclusive — production systems often combine them: prompt engineering for task framing, RAG for external knowledge, fine-tuning for domain adaptation. + +| Dimension | RAG | Fine-Tuning | Prompt Engineering | +|---|---|---|---| +| **Knowledge freshness** | Excellent (query live sources) | Poor (frozen at training time) | Depends on base model | +| **Training cost** | Low (no training needed) | High (GPU hours) | None | +| **Inference cost** | Medium (retrieval + generation) | Low (single model call) | Lowest | +| **Latency** | Higher (retrieval overhead) | Low | Lowest | +| **Factual accuracy** | High (grounded in retrieved docs) | Variable (model knowledge) | Variable | +| **Custom tone/behavior** | Limited (prompt guidance) | Excellent (learned patterns) | Moderate | +| **Data privacy** | Good (data stays in DB) | Risk (data in training set) | Good (no data storage) | +| **Handling new data** | Instant (add to index) | Requires retraining | N/A | +| **Long-term memory** | Excellent (index persists) | Good (learned patterns persist) | None | +| **Complex reasoning** | Good (augmented context) | Better (learned patterns) | Basic | +| **Explainability** | High (cite retrieved docs) | Low (black box) | Low | +| **Scalability** | Excellent (independent index) | Good (separate model per task) | Excellent | + +### Interview Questions +**Q: When would you combine RAG + fine-tuning?** +A: Common patterns: (1) Fine-tune for domain language understanding + RAG for specific facts. Example: a medical LLM fine-tuned on clinical notes (learns medical terminology, note format) with RAG on latest drug interactions, treatment guidelines. (2) Fine-tune to improve instruction following and output formatting, RAG for content. (3) Fine-tune a smaller model on task-specific data, use RAG to supplement knowledge — this achieves good quality with lower cost than a large model + RAG alone. + +**Q: What are the limitations of relying solely on prompt engineering?** +A: Prompt engineering is limited by: (1) Context window — can't provide enough information for complex tasks. (2) Consistency — prompt wording changes affect output. (3) Task complexity — can't teach the model new patterns the base model doesn't understand (e.g., a specific output schema requiring structured reasoning). (4) Memory — no long-term memory between sessions without retrieval augmentation. (5) Reliability — prompt injections and jailbreaks are easier to exploit. Prompt engineering is best for simple, well-defined tasks; RAG and fine-tuning extend its capabilities. + +--- + +## Additional Prompt Engineering + +**17. Role Prompting** +**Definition:** Assigning a specific persona or role to the LLM in the system prompt to set context, constraints, and behavioral expectations. +### Theory & Explanation +Role prompting frames the LLM's behavior by defining who it is, what expertise it has, and how it should respond. This activates domain-specific knowledge the model learned during training — a model prompted as "expert cardiologist" will surface medical knowledge more reliably than a generic "assistant." The role provides implicit guardrails: an "expert lawyer" knows to cite precedents and use precise legal language; a "creative writing coach" knows to focus on narrative techniques. Effective role prompts include: the role definition, the context/scenario, specific output guidelines, and constraints/tone. Role prompting is most effective for domain-specific tasks (medical, legal, technical) where the model has training data in that domain. It's less effective for novel domains (emerging fields not in training data) where the model cannot actually "be" an expert in something it wasn't trained on. +### Example +System: "You are an expert Python engineer with 15 years of experience. You prioritize readable, well-documented, type-annotated code. You explain performance implications and potential pitfalls." User: "Write a function to merge two sorted lists." → The model produces a type-annotated, documented function with time/space complexity analysis. Without role prompting, it might produce a generic unannotated solution. +### Interview Questions +**Q: How do you measure if role prompting improves output quality?** +A: Compare outputs with and without role prompting on representative test examples using: domain expert human evaluation (blinded), automated metrics (ROUGE-L, BERTScore for reference-based tasks), or LLM-as-judge evaluation. Track task-specific metrics: relevance, completeness, and adherence to domain conventions. A/B test in production on task success rate and user satisfaction. + +**Q: Can role prompting backfire?** +A: Yes — an overly specific or mismatched role can constrain the model: (1) The model might refuse to answer outside its role even when the answer is simple ("As a poet, I cannot help with mathematics"). (2) The model might exaggerate the role and produce verbose, stylistically rigid responses. (3) The role can create false confidence (an "expert" role may reduce self-correction). Solution: balance role specificity with flexibility: "You are an expert [role] but you can help with any question." + +--- + +**18. APE (Automatic Prompt Engineer)** +**Definition:** An automated method where an LLM generates, evaluates, and iteratively refines prompts to optimize performance on a given task. +### Theory & Explanation +APE (Zhou et al., 2022) treats prompt engineering as a search problem. The process: (1) Generate candidate prompts: use an LLM to produce many prompt variants given a task description (e.g., "Generate 20 prompts for sentiment classification"). (2) Evaluate candidates: run each prompt on a held-out validation set with labeled examples, measuring performance (accuracy, F1, or task-specific metric). (3) Select best: pick the top-performing prompt. (4) Iterate: use the best prompt to seed generation of refined candidates, repeating for several rounds. APE can discover prompts that outperform human-written ones, often with surprising formulations humans wouldn't try. Variants include: instruction-focused APE (generate instruction only), few-shot APE (generate both instruction and examples), and chain-of-thought APE (generate reasoning steps). APE is compute-intensive (requires many LLM calls for evaluation) but can be fully automated. +### Example +Task: Classify movie reviews as positive/negative. APE generates candidates: (1) "Classify the sentiment of this review." (Accuracy: 85%). (2) "You are a film critic. Is this review positive or negative?" (87%). (3) "Identify whether the reviewer liked or disliked the movie. Reply with 'Positive' or 'Negative' only." (91%). After 3 rounds of refinement, the best prompt: "Determine the sentiment expressed in the following movie review. Consider the overall tone, specific praise or criticism, and the reviewer's recommendation. Output only 'Positive' or 'Negative'." (Accuracy: 94% — vs human baseline 91%). +### Interview Questions +**Q: What are the limitations of APE?** +A: (1) Computationally expensive — evaluating 50 prompts on 100 examples requires 5000 LLM calls. (2) Metric-dependent — quality depends on having a good evaluation metric; for subjective tasks (creativity, style), metrics are hard to define. (3) Overfitting — prompts optimized on a small validation set may not generalize. (4) Search space — the best prompt may be outside the LLM's generation capability for prompts (the meta-prompt limits creativity). (5) Diminishing returns — improvement typically saturates after 2-3 rounds. + +**Q: How does APE compare with DSPy?** +A: APE optimizes prompts only — it searches the natural language instruction space. DSPy (Declarative Self-improving Python) is a broader framework that optimizes the entire pipeline: prompts, few-shot examples, tool calls, and retrieval parameters. DSPy treats pipeline steps as "modules" and uses Bayesian optimization or random search to find optimal configurations. APE is simpler and task-focused; DSPy is more powerful for complex multi-step pipelines but has a steeper learning curve. + +--- + +**19. Prompt Chaining** +**Definition:** Breaking a complex task into a sequence of simpler subtasks, where the output of each prompt becomes the input of the next. +### Theory & Explanation +Complex tasks often exceed what a single prompt can reliably handle. Prompt chaining decomposes the task into steps, each handled by a separate prompt (possibly with different instructions, contexts, and even different models). Each step's output feeds into the next. Advantages: (1) Modularity — each step is independently testable and optimizable. (2) Transparency — the chain's intermediate outputs provide insight into reasoning. (3) Specialization — different steps can use different models (a cheap model for extraction, a powerful model for reasoning). (4) Context management — each prompt can have focused context without exceeding token limits. Architectures: sequential chains (A → B → C), conditional chains (A → if X then B else C), and parallel chains (A and B in parallel → merge → C). The trade-off is increased latency (multiple LLM calls) and error propagation (a mistake early in the chain compounds). LLM-based evaluation can be inserted between steps for quality control. +### Example +Document summarization chain: Step 1: "Extract all key entities and their relationships from this document." → Entities list. Step 2: "Generate a one-paragraph summary focusing on the extracted entities." → Draft summary. Step 3: "Check if the draft summary covers all entities from step 1. If any entity is missing, expand the summary." → Final summary. Each step has a clear objective and limited context. +### Interview Questions +**Q: How do you decide the optimal granularity for chain steps?** +A: Each step should be a meaningful, well-defined subtask with a single clear output. Signs of too fine-grained: overhead dominates (latency of 10+ LLM calls for a 2-step task). Signs of too coarse: step prompts are complex and unreliable — decompose further. A good heuristic: each step should take 1-2 paragraphs of instructions and produce 1-2 paragraphs of output. Start with a 2-3 step decomposition and add steps only when a step's output quality is insufficient. + +**Q: How does prompt chaining compare to multi-step agents?** +A: Prompt chaining is a hard-coded sequence with fixed steps. Multi-step agents use an LLM to dynamically decide the next step, tool, and when to stop. Chaining is more predictable and debuggable; agents are more flexible but less reliable. Use chaining for well-understood, repeatable workflows (data processing pipelines, report generation). Use agents for exploratory or variable workflows (web research, customer support triage). + +--- + +**20. Common Prompt Templates** +**Definition:** Ready-made, reusable prompt patterns optimized for common NLP tasks, each with a standard structure and best practices. +### Theory & Explanation +Instead of writing prompts from scratch, common templates provide battle-tested formats for frequent tasks. Each template includes: system instruction, task description, input format specification, output format specification, and examples (few-shot). The templates serve as starting points that can be customized per use case. Common templates include: QA Template (context + question → answer), Summarization Template (document → bullet points or paragraph summary), Extraction Template (document + schema → structured output), Classification Template (text + categories → category label), Code Generation Template (requirements + language → code), Data Transformation Template (input format → output format, e.g., JSON to CSV). Effective templates encode specific heuristics: answer format constraints (JSON schema, markdown structure), length constraints (2-3 sentences, 100 words max), quality instructions (cite sources, explain reasoning), and edge case handling (how to handle insufficient information, conflicting sources). +### Example +**Extraction Template:** +``` +System: You are a data extraction expert. Extract structured information from the provided text according to the specified schema. Return valid JSON only. +User: Text: {input_text} +Schema: {json_schema} +Extract the data and return as JSON. +``` +**QA Template:** +``` +System: Answer the question based on the context below. If the context doesn't contain sufficient information, say "I don't have enough information to answer this." Do not make up facts. +Context: {retrieved_chunks} +User: {question} +Assistant: +``` +### Interview Questions +**Q: What makes a good prompt template versus a bad one?** +A: Good templates are: (1) Explicit about input/output format (no ambiguity). (2) Include examples (even 1-2 help dramatically). (3) Handle edge cases (what to do with missing information). (4) Use separators clearly (### Context: / ### Question: / ### Answer:). (5) Avoid negative instructions (don't say "Don't use bullet points" — say "Use numbered list"). Bad templates: (1) Overly vague. (2) No format specification. (3) Contradictory instructions. (4) Too long (exceed activation/attention span). (5) Role mismatches with task. + +**Q: How do you version and manage prompt templates in production?** +A: Treat prompts as code: (1) Store in version control (Git) with semantic versioning. (2) Each template has a unique name and version tag (e.g., "qa-template-v3"). (3) Maintain a changelog documenting what changed and why. (4) Run regression tests when updating template: compare outputs on a test set before/after. (5) Use feature flags to gradually roll out new templates (10% → 50% → 100% traffic). (6) Log which template version was used for each request for debugging. Tools like LangSmith, Weights & Biases Prompts, or simple YAML files work well. + +--- + +## Additional Agent Concepts + +**21. Reflection** +**Definition:** An agent evaluating its own output and iterating to improve it through self-critique, error detection, and plan revision. +### Theory & Explanation +Reflection gives agents a feedback loop: generate output → evaluate output → revise. This mirrors the human process of drafting and editing. The agent can evaluate against: correctness (does the output satisfy the task?), completeness (are there gaps?), coherence (is the reasoning logical?), constraints (does it follow format rules?). Reflection can be triggered: after each action (fine-grained), after completing a plan (checkpoint-based), or when an error or low-confidence indicator appears (on-demand). The evaluation can use: the same LLM (self-reflection), a different, more powerful model, or external tools (test runners, validators, linters). Reflection significantly improves agent performance — agents with reflection achieve 20-40% higher task completion rates on complex multi-step tasks. However, reflection adds latency (2x-3x more LLM calls) and can over-correct (make correct outputs worse by second-guessing). +### Example +Code agent generates a Python function to sort a list → reflects: "Does this handle empty lists? Edge case: what if input is None? Check time complexity — O(n²) from bubble sort. I should use Timsort instead." → Revises function with better implementation, type hints, and None handling. Then reflects again: "Test with input [3, 1, 4] → output [1, 3, 4]. Correct. Ready." +### Interview Questions +**Q: What are the failure modes of reflection?** +A: (1) Over-correction: correct output gets "corrected" to wrong output. Mitigation: limit reflection rounds (max 2-3). (2) Self-reinforcing errors: the model confirms its own mistake because it can't detect the error. Mitigation: use a different evaluator model. (3) Endless loops: agent keeps refining without convergence. Mitigation: max iteration limit, improvement threshold (stop if improvement < 5%). (4) Confirmation bias: the reflection prompt leads the model to approve its own output. Mitigation: neutral reflection prompts. + +**Q: How do you implement reflection in practice?** +A: Simple implementation: after each action, call the LLM with: "Given the original task: {task}, your previous output: {output}, evaluate the output. Identify any errors, omissions, or improvements. Then provide a revised version." Wrap in a loop with max_iterations=3. In LangGraph, reflection is a conditional edge: after generation → evaluation node → if "needs improvement" → loop back to generation. Track improvement metrics to detect diminishing returns. + +--- + +**22. Self-Evaluation** +**Definition:** An agent assigning a score or confidence rating to its own output, used to decide whether to stop, retry, switch strategies, or request human intervention. +### Theory & Explanation +Self-evaluation differs from reflection: reflection produces revised output; evaluation produces a score/decision. The model assesses: confidence in the answer, completeness against task requirements, alignment with constraints, and uncertainty about specific parts. Self-evaluation enables the agent to: (1) Stop early if sufficiently confident (saves compute). (2) Retry if confidence is low (improves quality). (3) Switch to a more capable model for hard cases. (4) Ask the human for clarification or approval. Self-evaluation can be: explicit (the model outputs a confidence score or explanation) or implicit (track token-level probabilities, entropy, or consistency across multiple generations). Calibration is key — a well-calibrated model knows when it's correct and when it's guessing. Most LLMs are overconfident — they assign high confidence even to wrong answers — so calibration techniques (temperature sampling, self-consistency) are critical. +### Example +Agent asked "What is the capital of France?" → Generates "Paris" with probability 0.92 → Self-evaluation: "Confidence: high. The answer is known and unambiguous." → Stop and return. Agent asked "What was the GDP of Argentina in 2023?" → Generates "Argentina's GDP in 2023 was approximately $640 billion" with probability 0.45 → Self-evaluation: "Confidence: medium. The specific number might not be current. I should verify with a web search." → Triggers retrieval before finalizing. +### Interview Questions +**Q: How do you calibrate an LLM's self-evaluation?** +A: Calibration involves aligning predicted confidence with actual accuracy: (1) Collect a calibration set of N examples with known ground truth. (2) For each example, have the model generate an answer + confidence score. (3) Bin examples by confidence (e.g., 0-0.2, 0.2-0.4, etc.) and compute accuracy per bin. (4) If accuracy < confidence in a bin (e.g., 90% confidence but 70% accuracy), the model is overconfident — apply Platt scaling or temperature scaling to adjust. (5) Iterate until calibration curve is diagonal. Post-hoc calibration (adjusting after generation) is more reliable than expecting the model to self-calibrate. + +**Q: How does self-evaluation enable selective delegation?** +A: Selective delegation uses self-evaluation to route tasks: if self-evaluation confidence > threshold (e.g., 0.8), use a small cheap model directly. If between 0.4-0.8, use a larger model. If < 0.4, use RAG + large model, or escalate to human. This cascade optimizes cost-quality trade-off: 60-80% of queries handled by the cheap model, 15-30% by the expensive model, 5-10% escalated to human. The thresholds are tuned based on task criticality. + +--- + +**23. Router** +**Definition:** A component that classifies an incoming query and routes it to the appropriate agent, tool, or processing pipeline. +### Theory & Explanation +Routers solve the problem of scale and specialization — instead of one giant agent that can do everything (expensive, error-prone), a router classifies the query and dispatches to specialized sub-agents. The router typically performs: intent classification (what does the user want?), entity extraction (what/who does the query refer to?), and urgency/priority assessment. Routing can be: (1) LLM-based: "Classify this query into one of: [categories]" — most flexible, higher latency. (2) Embedding-based: embed query, find nearest category centroid — faster, less flexible. (3) Rule-based: keyword/regex matching — fastest, brittle. Routers are the primary mechanism for reducing a monolithic system into manageable, specialized components. In production, routers handle load balancing, A/B testing (route to different system versions), and graceful degradation (route to fallback if primary system fails). Multi-level routing is common: first router (domain level: "tech support" vs "sales"), second router (sub-domain: "billing" vs "technical issue" vs "account management"). +### Example +Customer support system: Router receives "I can't log in to my account." → Classifies as "Authentication Issue" → Routes to Password Reset Agent. Router receives "What's the price of the premium plan?" → Classifies as "Pricing Question" → Routes to Sales Info Agent (which retrieves latest pricing from DB). Router receives "You guys are terrible, I want a refund" → Classifies as "Refund/Churn" → Routes to Retention Specialist Agent (with high urgency flag). +### Interview Questions +**Q: How do you train a router?** +A: Three approaches: (1) LLM-based: craft a system prompt with clear category definitions and examples. Zero-shot works well for 5-15 categories. Use few-shot for rare categories. (2) Embedding + classifier: collect labeled examples (query → category), embed queries with sentence transformer, train a lightweight classifier (logistic regression, SVM) on the embeddings. (3) Fine-tuned classifier: fine-tune a small BERT-style model on labeled queries. LLM-based is easiest to start; embedding-based is best for latency-critical systems; fine-tuned is best for high accuracy with many fine-grained categories. In production, start with LLM and move to embedding/classifier as query volume grows. + +**Q: What happens when the router is uncertain?** +A: Strategies: (1) Top-2 routing — run the top-2 most likely agents and fuse or compare results. (2) Confidence threshold — if max confidence < 0.6, use a general-purpose fallback agent or ask the user for clarification. (3) Explicit disambiguation — router says "Did you mean [option A] or [option B]?" (4) Escalate to human — for highly uncertain or high-stakes queries. The fallback behavior should be designed to gracefully handle uncertainty rather than making a wrong hard routing decision. + +--- + +**24. Autonomous vs Semi-autonomous Agents** +**Definition:** Autonomous agents operate without human intervention; semi-autonomous agents pause for human confirmation on key decisions. +### Theory & Explanation +The autonomy spectrum ranges from fully autonomous (AI plans, executes, and completes tasks independently) to fully human-operated (AI only provides suggestions). Semi-autonomous agents sit in the middle: they can perform routine sub-tasks independently but require human approval for high-stakes actions (sending emails, making purchases, posting content, deleting data). The decision boundary is determined by: (1) Risk: irreversible actions (deleting data, spending money) need approval; reversible ones (searching, drafting) can be autonomous. (2) Cost: expensive operations (buying cloud resources) need approval; cheap ones (API calls) don't. (3) Compliance: regulated actions (medical advice, legal decisions) require human oversight. (4) User preference: some users want more control, others want automation. Semi-autonomous agents typically implement: draft mode (agent acts, asks "Should I proceed?"), approval queues (proposed actions displayed for batch approval), and confidence thresholds (agent proceeds autonomously only when confidence > threshold). +### Example +Autonomous email assistant: reads email, categorizes, auto-replies to routine messages (scheduling, confirmations), flags complex emails for human review. Semi-autonomous email assistant: drafts replies to all emails, presents drafts to user for review and approval before sending. For the semi-autonomous version, the user can: approve (send), edit then approve, reject and write from scratch, or set rules (auto-approve emails from specific senders). +### Interview Questions +**Q: How do you decide the autonomy level for a given agent?** +A: Assess each action the agent can take on two dimensions: (1) Reversibility: is the action undoable? (Deleting a file → undoable. Searching → reversible.) (2) Impact: what's the potential negative consequence? (Low: minor inconvenience. High: financial loss, compliance violation, user trust damage.) Rule: high-impact irreversible actions require always-confirm. Low-impact reversible actions can be autonomous. Actions in the middle use confidence thresholds or user preference settings. + +**Q: What user experience patterns work for semi-autonomous agents?** +A: (1) Notification + quick action: "I've drafted a reply. [Approve] [Edit] [Reject]" — shown inline. (2) Approval dashboard: list of pending actions with approve/reject/edit for each. (3) Gradual autonomy: start with maximum human oversight, reduce based on user's trust patterns (if user approves 10 emails from the same sender without changes, start auto-approving). (4) Undo: even for autonomous actions, provide a brief undo window (e.g., Gmail undo send). (5) Audit log: always show what the agent did, even if autonomous. + +--- + +**25. Agent Architecture** +**Definition:** Common structural patterns for organizing LLM agents, defining how agents communicate, delegate, and collaborate. +### Theory & Explanation +Agent architectures solve coordination: when you have multiple capabilities, how do you organize them? Four dominant patterns: (1) Single-agent: one LLM + tools. Simplest, good for well-scoped tasks. The LLM decides tool usage, call ordering, and final output. (2) Supervisor/Worker: an orchestrator (supervisor) delegates subtasks to specialized worker agents, collects results, and composes the final output. Workers are domain-specific (code generator, web searcher, data analyzer). The supervisor handles planning and coordination. (3) Multi-agent: multiple agents with different roles collaborate without a single supervisor. Agents communicate via shared messages, each contributing from their expertise. Requires coordination protocols to avoid conflicts. (4) Hierarchical: multi-level decomposition — strategic agents set goals, tactical agents plan approaches, operational agents execute. Each level has different scope and authority. The choice depends on: task complexity (single vs multi-step), domain breadth (single vs multiple domains), need for specialization, and coordination complexity. +### Example +**Supervisor/Worker** for report generation: Supervisor receives "Research and write a report on AI regulation." → Supervisor splits into: Research Worker (search for articles, papers), Analysis Worker (extract key themes), Writing Worker (compose report sections), Citation Worker (format citations). Supervisor collects outputs, composes final report, checks quality. +### Interview Questions +**Q: When should you use multi-agent over supervisor/worker?** +A: Multi-agent is better when: (1) No natural hierarchy — agents have equal status (debate, discussion, collaborative problem-solving). (2) Emergent behavior is desired — interaction between agents generates novel solutions. (3) Resilience — no single point of failure. Supervisor/worker is better when: (1) Clear task decomposition is possible. (2) Centralized coordination is needed (prevent conflicts). (3) Simpler debugging and logging. In practice, supervisor/worker is more common because it's easier to reason about and control. + +**Q: What are the failure modes of multi-agent systems?** +A: (1) Coordination overhead: agents spend more time communicating than doing useful work. (2) Conflicts: agents give contradictory instructions or overwrite each other's outputs. (3) Deadlocks: agent A waits for B, B waits for A. (4) Echo chamber: agents reinforce each other's incorrect assumptions. (5) Cost explosion: N agents × multiple rounds = high token usage. Solutions: timeouts, conflict resolution protocols, termination conditions, and shared memory for grounding. + +--- + +**26. LangChain** +**Definition:** A popular open-source framework for building LLM-powered applications, providing abstractions for agents, chains, tools, memory, and retrieval. +### Theory & Explanation +LangChain (2022, Harrison Chase) standardizes the components needed for LLM applications: Models (abstraction layer over different LLM providers), Prompts (templates, few-shot management), Chains (composable sequences of calls), Agents (LLM-driven tool usage with ReAct or Plan-and-Execute patterns), Tools (wrappers for APIs, search, code execution, databases), Memory (conversation history, entity summaries, vector store retrieval), and Retrieval (document loaders, text splitters, embedding, vector stores). LangGraph extends LangChain with state-graph architecture: nodes (processing steps) and edges (conditional transitions between nodes) that enable complex, cyclic workflows (multi-agent loops, reflection, human-in-the-loop). LangSmith provides observability: tracing, evaluation, and debugging. LangChain is framework-agnostic (works with any LLM) but introduces framework-specific abstractions that can leak abstraction and make debugging harder. +### Example +Building a RAG agent in LangChain: `vectorstore = Chroma.from_documents(docs, embeddings)` → `retriever = vectorstore.as_retriever()` → `chain = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)` → `chain.invoke("What is RAG?")`. Adding tools: `tools = [TavilySearchResults(), PythonREPL()]` → `agent = create_react_agent(llm, tools)` → execute. +### Interview Questions +**Q: What are the pros and cons of using LangChain vs building from scratch?** +A: Pros: rapid prototyping, built-in components for common patterns (RAG, agents, chains), provider agnosticism, community extensions, observability via LangSmith. Cons: abstraction overhead (complex stack traces, hard to debug), framework lock-in, version instability (breaking changes between minor versions), "black box" feeling (hard to know exactly what's happening under the hood). Recommendation: use LangChain for prototyping and standard patterns; consider building on plain LLM APIs for production when you need full control over prompt construction, error handling, and latency. + +**Q: How does LangGraph differ from LangChain?** +A: LangChain is linear/compositional — chains execute in sequence. LangGraph is graph-based — nodes execute in a directed graph with cycles, conditionals, and parallel execution. LangGraph enables: loops (reflection), branching (parallel tool calls), state machines (human-in-the-loop), and multi-agent orchestration. Think of LangChain as "deterministic pipeline" and LangGraph as "stateful workflow." LangGraph subsumes LangChain — you can use LangChain components within a LangGraph. + +--- + +**27. Semantic Kernel** +**Definition:** Microsoft's lightweight SDK for AI orchestration, providing plugins, planners, and memory for building LLM applications in .NET and Python. +### Theory & Explanation +Semantic Kernel (SK) is designed for enterprise integration — it emphasizes type safety, native code integration, and enterprise patterns. Core concepts: (1) Kernel: central orchestrator that connects LLMs, plugins, and memory. (2) Plugins: reusable, typed functions (native code or semantic prompts) that the AI can call. Plugins have explicit schemas (input/output types) for safety. (3) Planner: automatically sequences plugin calls to fulfill a user request. SK's planner uses function composition and can handle multi-step plans. (4) Memory: semantic memory (vector storage for facts), text memory (text-based retrieval). SK is .NET-native (C#/F#) with Python support — it integrates with Azure OpenAI, OpenAI, Hugging Face, and local models. Compared to LangChain, SK is more opinionated about structure (types, schemas) and enterprise security, but has a smaller community and fewer integrations. +### Example +C# code: `var kernel = Kernel.CreateBuilder().AddAzureOpenAIChatCompletion(deploymentName, endpoint, apiKey).Build(); kernel.ImportPluginFromType(); var result = await kernel.InvokePromptAsync("Send an email to {{$recipient}} about {{$topic}}", new() { ["recipient"] = "user@example.com", ["topic"] = "meeting reminder" });` The planner automatically calls the EmailPlugin's SendEmail function with the right parameters. +### Interview Questions +**Q: How does Semantic Kernel's planner compare to LangChain agents?** +A: SK's planner (FunctionCallingStepwisePlanner) uses a sequential planning approach — it generates a step-by-step plan as a sequence of function calls, then executes them. LangChain agents use ReAct (Reasoning + Acting loop) — interleave reasoning and tool calls dynamically. SK's planner is more structured (the plan is generated upfront); LangChain's agents are more flexible (plan can change based on intermediate results). SK's planner is better for well-defined, multi-step workflows; LangChain agents are better for exploratory tasks. + +**Q: When would you choose Semantic Kernel over LangChain?** +A: Choose Semantic Kernel when: (1) Your tech stack is .NET/C# (SK is the natural choice). (2) Enterprise security and type safety are priorities (SK's typed plugins prevent malformed calls). (3) You want tight Azure OpenAI integration. (4) You prefer structured planning over dynamic agent behavior. Choose LangChain when: (1) Your stack is Python/JavaScript. (2) You need the largest ecosystem of integrations. (3) You're building exploratory or research-oriented applications. (4) You need multi-agent systems (SK's multi-agent support is less mature). + +--- + +## Additional Graph ML & KG Concepts + +**28. RDF (Resource Description Framework)** +**Definition:** W3C standard data model for representing knowledge graphs as triples (subject, predicate, object), enabling interoperable data exchange. +### Theory & Explanation +RDF models all knowledge as statements in the form (subject, predicate, object) — "The sky has color blue" becomes (Sky, hasColor, Blue). Subjects and predicates are URIs (globally unique identifiers). Objects can be URIs (for entities) or literals (strings, numbers, dates). This simple triple structure makes RDF highly extensible — anyone can add statements about any resource using any vocabulary. Multiple serialization formats exist: Turtle (compact, human-readable), RDF/XML (machine-readable, integrates with XML toolchains), JSON-LD (JSON-based, web-friendly), and N-Triples (line-based, easy to parse). RDF enables data integration across sources — different organizations can publish RDF data using shared or mapped vocabularies, and SPARQL queries can span multiple datasets. The linked data principles extend RDF: use HTTP URIs for identifiers, provide useful information at those URIs, and include links to related URIs. +### Example +Turtle serialization: `@prefix foaf: . @prefix ex: . ex:Alice a foaf:Person ; foaf:name "Alice" ; foaf:knows ex:Bob . ex:Bob foaf:name "Bob" .` This expresses: Alice is a person, her name is "Alice", she knows Bob, and Bob's name is "Bob". All identifiers are URIs making this publishable on the web. +### Interview Questions +**Q: What are the strengths and weaknesses of RDF compared to property graphs?** +A: RDF strengths: (1) Web-native — uses URIs, enabling global interoperability and data linking across sources. (2) Standards-based — W3C standards (RDFS, OWL, SPARQL) ensure long-term stability. (3) Open-world assumption — new statements can always be added without schema changes. Weaknesses: (1) Verbosity — URIs make RDF files large and hard to read. (2) No built-in property mechanism for edges (reification is complex). (3) Query performance — SPARQL is less efficient than Cypher/GraphQL for graph traversals. Property graphs are simpler, more performant for graph algorithms, and more intuitive for developers, but lack web-native interoperability. + +**Q: What is SPARQL and how does it relate to RDF?** +A: SPARQL (SPARQL Protocol and RDF Query Language) is the query language for RDF data. It uses graph pattern matching — write patterns with variables, and the engine finds all matching subgraphs. Example: `SELECT ?name WHERE { ?person foaf:name ?name . ?person foaf:knows ex:Alice }` — find names of people who know Alice. SPARQL supports federation (query across multiple RDF datasets), graph construction (create new RDF graphs from query results), and update operations (INSERT/DELETE). SPARQL is to RDF as SQL is to relational databases. + +--- + +**29. RDFS/OWL** +**Definition:** Ontology languages for defining the schema and logical constraints of knowledge graphs expressed in RDF. +### Theory & Explanation +RDF Schema (RDFS) provides basic ontology building blocks: classes (rdfs:Class), class hierarchies (rdfs:subClassOf — "Dog is a subclass of Animal"), properties (rdfs:Property), property hierarchies (rdfs:subPropertyOf), domain/range constraints (rdfs:domain, rdfs:range — "hasOwner property has domain Pet and range Person"). RDFS enables basic inference: if Dog subClassOf Animal, and Fido is a Dog, then Fido is also an Animal. OWL (Web Ontology Language) extends RDFS with expressive logical constraints: equivalence (owl:equivalentClass — "Person is equivalent to Human"), disjointness (owl:disjointWith — "Cat and Dog are disjoint"), property characteristics (owl:transitiveProperty — "isAncestorOf" is transitive; owl:inverseOf — "owns ⇔ isOwnedBy" are inverses; owl:functionalProperty — a person has exactly one birth date), cardinality restrictions (owl:cardinality — "exactly 2 parents"), and complex class expressions (owl:unionOf, owl:intersectionOf, owl:complementOf). OWL reasoners (Pellet, HermiT) can infer implicit knowledge, detect inconsistencies, and answer queries involving complex logical relationships. +### Example +RDFS: `ex:Vehicle rdfs:subClassOf ex:Machine . ex:Car rdfs:subClassOf ex:Vehicle . ex:hasSpeed rdfs:domain ex:Vehicle ; rdfs:range xsd:integer . ex:MyCar a ex:Car ; ex:hasSpeed 120 .` → Inference: MyCar is a Vehicle and also a Machine. OWL addition: `ex:Car owl:disjointWith ex:Boat . ex:Motorboat a ex:Boat .` → Reasoner can infer that ex:MyCar is not a Motorboat. +### Interview Questions +**Q: What is the trade-off between RDFS and OWL expressivity?** +A: RDFS is lightweight and efficient — basic class hierarchies and property constraints can be processed quickly, but the reasoning is limited. OWL 2 DL (Description Logic profile) provides rich expressivity (all relationship types) but reasoning is computationally expensive (worst-case NEXPTIME-complete). OWL 2 RL (Rule-based profile) provides a practical subset that can be implemented with rule engines (faster but less expressive). Choose RDFS when you need simple vocabulary organization. Choose OWL 2 RL for practical reasoning at scale. Choose OWL 2 DL when you need maximum expressivity and can tolerate slower reasoning. + +**Q: How do ontologies differ from database schemas?** +A: Database schemas define structure with constraints but don't support inference — if you add a new category (e.g., "SUV is-a Car"), you must update the schema and rewrite queries. Ontologies are open-world: new classes, properties, and instances can be added without schema changes, and reasoners automatically infer new relationships. Database schemas use closed-world assumption (what's not in the DB is false); ontologies use open-world (what's not in the KG is unknown). This makes ontologies more flexible for evolving domains but harder to guarantee completeness. + +--- + +**30. Node Embedding** +**Definition:** Unsupervised methods that learn low-dimensional vector representations of nodes in a graph by preserving the graph's structural properties. +### Theory & Explanation +Node embedding methods learn to map each node to a vector such that structurally similar nodes have similar vectors. DeepWalk (2014) pioneered random-walk-based embedding: (1) Run short random walks from each node (simulating graph exploration). (2) Treat each walk as a "sentence" of node IDs. (3) Apply Word2Vec (skip-gram) to learn embeddings that predict context nodes within the walk. Node2Vec (2016) extends DeepWalk with biased random walks controlled by two parameters: p (return parameter — high p avoids backtracking) and q (in-out parameter — high q focuses on local structure, low q explores outward). p > 1 + q < 1 = BFS-like (local neighborhoods). p < 1 + q > 1 = DFS-like (global structure). By tuning p and q, Node2Vec can capture different structural roles (hubs, bridges, periphery nodes). +### Example +In a social network graph: Node2Vec with p=1, q=0.5 → walks explore outward (DFS-like) → embeddings capture community structure (users in the same community have similar embeddings). Node2Vec with p=0.5, q=2 → walks stay local (BFS-like) → embeddings capture structural roles (two otherwise unconnected users who are both bridges between communities have similar embeddings). +### Interview Questions +**Q: How do node embeddings differ from GNN node representations?** +A: Node embeddings (DeepWalk, Node2Vec) are unsupervised and use only graph structure (edges). They don't incorporate node features, making them applicable to any graph but missing rich attribute information. GNN node representations (GCN, GAT) are typically supervised/semi-supervised and combine graph structure with node features (text, images, numerical attributes). GNNs are more expressive (can learn task-specific representations) but require labeled data and are more expensive. In practice: use node embeddings for unsupervised tasks (exploration, clustering) when you have just a graph; use GNNs for supervised tasks with rich node features. + +**Q: How does the choice of p and q in Node2Vec affect embeddings?** +A: p (return parameter): low p → walk tends to backtrack, exploring local neighborhood (BFS-like). High p → walk avoids revisiting nodes, explores outward (DFS-like). q (in-out parameter): low q → walk prefers outward exploration (captures community structure). High q → walk stays near start node (captures structural equivalence — nodes with similar neighborhood patterns). Practically: for community detection → low q (0.5-1). For role detection (identifying hubs, bridges) → high q (2-4). For general tasks → p=1, q=1 (unbiased). + +--- + +**31. Graph Embedding** +**Definition:** Methods that encode an entire graph or subgraph into a fixed-size vector representation, enabling graph-level prediction tasks. +### Theory & Explanation +While node embeddings represent individual nodes, graph embeddings represent entire graphs (collections of nodes and edges). This is essential for tasks like molecular property prediction (predict toxicity of a molecule from its graph structure). Graph2Vec (2019) extends the Node2Vec approach to graph-level: (1) Extract rooted subgraphs (WL kernel features) from all nodes across all graphs. (2) Treat each graph as a "document" of its subgraph features. (3) Apply Doc2Vec to learn graph embeddings. GL2Vec uses the same approach but also considers node and edge labels. More modern approaches use GNNs with readout functions: apply GNN layers to get node representations, then pool (mean, max, attention) to get graph representation. Graph-level embeddings are used for graph classification, similarity search (find similar molecules), and graph generation evaluation. +### Example +Molecular property prediction: 1000 molecules. Represent each molecule as a graph (atoms = nodes, bonds = edges). Graph2Vec embeds each molecule to a 128-dim vector. Train a classifier (random forest, SVM) on these vectors to predict "toxic or not" with 85% accuracy. Alternative: use a GNN with global mean pooling — end-to-end training achieves 92% but requires labeled data and compute. +### Interview Questions +**Q: How do graph embeddings differ from graph kernels?** +A: Graph kernels (WL kernel, shortest-path kernel, random walk kernel) measure similarity between graphs by comparing their substructures without producing explicit embeddings. They are interpretable (you can see which substructures match) but don't scale well (O(n²) graph comparisons). Graph embeddings produce explicit vectors (O(n) comparisons after embedding), enabling standard ML pipelines (SVM, neural nets). Modern approaches blur the line — GNNs with readout can be seen as learned, differentiable graph kernels. + +**Q: When would you use graph-level embeddings vs node-level aggregation?** +A: Graph embeddings (Graph2Vec, GL2Vec) are unsupervised — you don't need labeled data to train them, but they may miss task-specific patterns. Node-level aggregation (GNN + readout) is supervised — you train end-to-end for the specific task, achieving better accuracy when labels are available. Use graph embeddings when: (1) You don't have labeled data. (2) You need a fixed-size representation for any downstream task. (3) You're doing exploratory analysis (clustering graphs). Use GNN + readout when: (1) You have labeled data. (2) Your task is well-defined. (3) You can tolerate training time. + +--- + +**32. Graph Transformer** +**Definition:** Adapting the transformer architecture to graph-structured data by incorporating graph-specific positional encodings and attention mechanisms. +### Theory & Explanation +Graph Transformers apply the successful transformer architecture to graphs. The key challenge: transformers process sequences with explicit position information (sinusoidal positional encoding), but graphs lack a natural ordering. Solutions include: (1) Laplacian positional encoding: use eigenvectors of the graph Laplacian as position features — nodes with similar neighborhoods get similar position encodings. (2) Random walk structural encoding: encode structural roles based on landing probabilities of random walks. (3) Spatial encoding: encode shortest path distances between node pairs. Graph Transformers often outperform GNNs on molecular property prediction, especially for large graphs where GNN over-smoothing (all node representations converging) is a problem. Notable models: Graphormer uses centrality encoding (degree), spatial encoding (shortest path distance), and edge encoding (edge features in attention bias). TokenGT treats each node and edge as independent tokens with structural encoding. The main trade-off: Graph Transformers have O(N²) attention cost vs GNNs' O(N + E) — they're more expressive but less scalable. +### Example +On the ZINC molecular property prediction benchmark: Graphormer achieves MAE 0.122 (vs GNN baseline 0.25-0.35). For a molecule graph with N=50 atoms, the transformer computes 50×50 attention, capturing long-range interactions between distant atoms that GNNs (limited by message-passing depth) would miss. +### Interview Questions +**Q: When are Graph Transformers preferable to GNNs?** +A: Graph Transformers excel when: (1) Long-range dependencies matter — two distant nodes interact meaningfully (molecular property prediction where distant functional groups interact). (2) Graphs are moderate in size (N < 5000, where O(N²) attention is feasible). (3) Node features are rich and relationships are complex. GNNs are better for: (1) Large graphs (millions of nodes). (2) Inductive settings (generalizing to unseen graphs, where GNNs' local message passing generalizes better). (3) Resource-constrained environments. In practice, hybrid approaches (GNN + local attention) often work best. + +**Q: What are the different positional encoding strategies for Graph Transformers?** +A: (1) Laplacian PE: eigenvectors of graph Laplacian — captures global graph structure but is sensitive to graph perturbations (adding/removing a node changes eigenvectors). (2) Random walk PE: based on random walk landing probabilities — captures structural roles, more stable. (3) Spatial PE: shortest path distances to a set of anchor nodes. (4) Degree PE: one-hot encoding of node degree — simple but limited. (5) SignNet: learns sign-invariant representations of eigenvectors (addressing sign ambiguity). The choice depends on whether you need global position (Laplacian) or local structure (Random walk PE). + +--- + +**33. RGCN (Relational Graph Convolutional Network)** +**Definition:** Extension of Graph Convolutional Networks to heterogeneous graphs with multiple relation types, using separate weight matrices per relation. +### Theory & Explanation +Standard GCNs assume homogeneous graphs (one relation type). RGCN extends GCNs to heterogeneous graphs where edges have types (e.g., "friend_of", "works_at", "located_in"). The core idea: each relation type r has its own weight matrix W_r. Node i's representation is updated by aggregating over its neighbors, but the aggregation is relation-specific. For graphs with many relation types (e.g., knowledge graphs with hundreds of relation types), independent weight matrices per relation lead to massive parameter counts and overfitting. Two parameter sharing techniques address this: (1) Basis decomposition: W_r = Σ_b a_{rb} V_b — each relation's weight is a linear combination of a small set of basis matrices V_b. (2) Block-diagonal decomposition: W_r is block-diagonal, each block processes a subspace of features. RGCN achieves state-of-the-art on link prediction and node classification in heterogeneous graphs. +### Example +A knowledge graph with relations: (Paris, capitalOf, France), (Alice, worksAt, CompanyX), (Alice, friendOf, Bob). RGCN layer: Alice's representation aggregates from: Bob (friend relation, using W_friend), CompanyX (worksAt relation, using W_worksAt). The model learns different transformations for different relationship types — friend influence might propagate personal interests, while work influence propagates professional skills. +### Interview Questions +**Q: How does RGCN handle the problem of many relation types?** +A: With N relations and D-dimensional features, RGCN needs N × D × D parameters for the weight matrices. For 100 relations and 512 dimensions, that's 100 × 512² ≈ 26M parameters per layer — prohibitively large. Solutions: (1) Basis decomposition: learn B basis matrices (B << N, typically 10-50) and N sets of combination coefficients — total params: B × D² + N × B. (2) Block-diagonal matrices: each W_r is block-diagonal with B blocks of size D/B — reduces params from D² to B × (D/B)² = D²/B. Both techniques reduce params by 10-100x while maintaining reasonable expressivity. + +**Q: When would you use RGCN vs simpler approaches for heterogeneous graphs?** +A: Use RGCN when: (1) There are multiple relation types (3+). (2) Each relation type likely requires different transformation patterns. (3) You have sufficient labeled data to train the many parameters. Use simpler approaches when: (1) Only 1-2 relation types — just use standard GCN. (2) Relations can be reduced to a single type (e.g., treat all edges the same). (3) Data is limited — simpler models (node embedding + classifier) may outperform RGCN. + +--- + +**34. KGE (Knowledge Graph Embedding)** +**Definition:** Methods that embed entities and relations in a knowledge graph into low-dimensional vector spaces, representing graph structure through geometric operations. +### Theory & Explanation +Knowledge graphs store facts as (head, relation, tail) triples. KGE models learn embeddings such that the triple's plausibility score is high for true facts and low for false ones. Three landmark models: (1) TransE (Bordes et al., 2013): interpret relations as translations in embedding space — h + r ≈ t for a true triple. Score function: f(h,r,t) = -||h + r - t||. Simple, efficient, works well for 1-to-1 relations but struggles with 1-to-N, N-to-1, and symmetric relations. (2) ComplEx (Trouillon et al., 2016): use complex-valued embeddings (real + imaginary parts). The score uses Hermitian dot product, which captures symmetric and antisymmetric patterns naturally. (3) RotatE (Sun et al., 2019): model relations as rotations in complex space — t = h ◦ r (element-wise rotation by angle θ_r). Can model symmetric (θ_r = 0 or π), antisymmetric (θ_r ≠ 0, π), inverse (r₂ = -r₁), and compositional patterns. +### Example +TransE: (Einstein, bornIn, Germany) → vector(Einstein) + vector(bornIn) ≈ vector(Germany). (Germany, capitalOf, Berlin) → vector(Germany) + vector(capitalOf) ≈ vector(Berlin). Composition: vector(Einstein) + vector(bornIn) + vector(capitalOf) ≈ vector(Berlin) → captures that Einstein was born in the capital of Germany. RotatE: for relation (marriedTo) which is symmetric (A marriedTo B ⇔ B marriedTo A), RotatE learns rotation θ=0 (identity) — embedding(A) ≈ embedding(B) after rotation. +### Interview Questions +**Q: Compare TransE, ComplEx, and RotatE — strengths and weaknesses.** +A: TransE: simplest and fastest, works well for 1-to-1 relations in relatively clean KGs. Fails on symmetric relations and 1-to-N relations. ComplEx: handles symmetric/antisymmetric patterns naturally via complex embeddings, better on complex KGs. Higher computational cost. RotatE: most expressive — can model symmetry, antisymmetry, inversion, and composition in a unified framework. Best overall performance but highest training cost. Rule of thumb: start with TransE for clean, simple KGs; use RotatE for complex, real-world KGs. + +**Q: How do you evaluate KGE models?** +A: Standard evaluation on link prediction: remove some triples from the KG, train on remaining, predict held-out triples. For each test triple (h,r,t), replace tail with all entities, rank by score. Metrics: Mean Reciprocal Rank (MRR — average of 1/rank across all test triples), Hits@K (fraction of test triples where correct entity is in top K — Hits@1, Hits@3, Hits@10). Key: filter setting — remove other valid triples from ranking. Filtered metrics are standard. Competitive scores on FB15k-237: MRR ~0.35, Hits@10 ~0.55 for RotatE. + +--- + +**35. Link Prediction** +**Definition:** The task of predicting missing edges in a graph based on existing structure and node/edge features. +### Theory & Explanation +Link prediction asks: given a graph with some observed edges, which unobserved edges are likely to exist? This is fundamental for recommendation systems (predict user-item interactions), social networks ("people you may know"), biological networks (predict protein-protein interactions), and knowledge graph completion (predict missing facts). The standard setup: (1) Partition edges into training (observed), validation, and test (held-out). (2) Train a model to score the likelihood of an edge between any pair of nodes. (3) Evaluate on held-out edges. Approaches range from simple heuristics to deep learning: heuristics (Jaccard similarity, Adamic-Adar, preferential attachment), node embedding methods (compute similarity between learned embeddings), GNN-based methods (score edge from node representations), and KGE methods (score triples in KGs). Evaluation metrics: MRR, Hits@K, ROC-AUC. Negative sampling is critical — randomly sampled non-edges as negative examples, typically 10-100 negatives per positive. +### Example +Social network: 1000 users, 5000 known friendships. Train to predict missing friendships. Node2Vec embeddings for each user. For users A and B, friendship score = cosine similarity(embed(A), embed(B)). Top predictions: A-B: 0.95 (they share 10 mutual friends), A-C: 0.91 (in same community), A-D: 0.12 (no mutual friends, different communities). Evaluation on 500 held-out friendships: Hits@10 = 0.72. +### Interview Questions +**Q: What are the evaluation pitfalls in link prediction?** +A: (1) Time leakage: using future interactions to predict past — always split chronologically for temporal graphs. (2) Degree bias: high-degree nodes are easier to predict — report metrics stratified by node degree. (3) Negative sampling bias: random negative samples are too easy — use harder negatives (e.g., negative sampling with node degree weighting). (4) Cold start: new nodes with few connections are hard to predict — evaluate separately. + +**Q: Compare heuristic-based vs learned link prediction.** +A: Heuristic methods (Jaccard, Adamic-Adar) are: fast (no training), interpretable, and strong baselines — often within 10-15% of learned methods. Learned methods (node embedding, GNN, KGE): more accurate, adaptable, and task-specific. Practical advice: always compute heuristic baselines first; if they're sufficient, stop. If not, use heuristics as features in a learned model. + +--- + +**36. Node Classification** +**Definition:** Predicting the label of a node given the graph structure and (optionally) node features, typically in a semi-supervised setting where only a few nodes have labels. +### Theory & Explanation +Node classification is the most common graph ML task. The key challenge is semi-supervised learning — using the graph structure to propagate information from labeled to unlabeled nodes. Standard methods: (1) GCN: aggregate neighbor features with normalized adjacency. (2) GAT: learn attention weights to determine which neighbors matter more. (3) GraphSAGE: sample neighbors (instead of using all), making it scalable. Training is typically transductive (for GCN/GAT) or inductive (GraphSAGE). Evaluation: accuracy, F1-score. Standard datasets: Cora (2708 nodes, 7 classes), Pubmed (19717 nodes, 3 classes), ogbn-arxiv (169343 nodes, 40 classes). State-of-the-art accuracy on Cora: ~85-88%. +### Example +Cora citation network: nodes = papers, edges = citations, features = bag-of-words of paper abstracts, labels = research area (7 classes). GCN with 2 layers (16 hidden dimensions) trained on 20 labeled papers per class (140 total), predicts labels for the remaining 2568 papers. GCN achieves ~82% accuracy. Key insight: the graph structure helps — a paper about RL that cites other RL papers will be classified as RL even if its abstract uses ambiguous terms. +### Interview Questions +**Q: How does semi-supervised node classification work?** +A: The graph provides a smoothness prior — connected nodes likely share labels. During training, the GCN loss is computed only on labeled nodes (cross-entropy). But message-passing ensures that labeled nodes influence their neighbors' representations, and those neighbors influence further neighbors. After 2-3 GCN layers, the model's receptive field covers the entire graph, so every node's representation has been shaped by labeled nodes. This is transductive learning. + +**Q: What are the limitations of GCN for node classification?** +A: (1) Over-smoothing: with many layers (4+), node representations converge to the same value. GCNs work best with 2-3 layers. (2) Homophily assumption: GCNs assume connected nodes have similar labels. On heterophilic graphs (where connected nodes have different labels), GCNs perform poorly. (3) Transductive: standard GCN can't predict labels for new nodes added after training. (4) Feature importance: GCN doesn't easily reveal which features drive predictions. + +--- + +**37. Graph Classification** +**Definition:** Predicting a single label for an entire graph, learning from a dataset of labeled graphs. +### Theory & Explanation +Graph classification assigns a label to each graph as a whole. The standard pipeline: (1) Apply GNN layers to learn node representations within each graph. (2) Apply a readout/pooling function to aggregate all node representations into a single graph representation. Common readout functions: mean pooling, max pooling, sum pooling (captures graph size), attention pooling, hierarchical pooling (DiffPool). (3) Pass the graph representation through an MLP classifier. Applications: molecular property prediction (predict toxicity from molecular graph), bioinformatics (predict protein function), and cheminformatics. Standard datasets: MUTAG (188 graphs, 2 classes), PROTEINS (1113 graphs, 2 classes), ogbg-molhiv (41127 graphs). +### Example +Molecule: benzene (C₆H₆) as a graph — 6 carbon nodes (connected in a ring), 6 hydrogen nodes. GNN processes the graph, readout produces embedding. Classifier: is it mutagenic? Trained on 150 labeled molecules, predicts for 38 held-out. GNN achieves ~85% accuracy vs ~80% for fingerprint-based baselines. +### Interview Questions +**Q: How does graph classification differ from node classification?** +A: Node classification predicts per-node labels using global graph structure. Graph classification predicts a global label per graph — the model must learn to ignore graph-irrelevant details and focus on label-relevant subgraphs. Different pooling strategies encode different inductive biases: sum pooling captures graph size, mean pooling is size-invariant, attention pooling learns which nodes are task-relevant. Graph classification requires multiple training graphs (typically 100+). + +**Q: What are the limitations of GNN-based graph classification?** +A: (1) Expressive power: standard GNNs are at most as powerful as the WL test — they can't distinguish certain non-isomorphic graphs. (2) Global structure: GNNs with few layers see only local neighborhoods; many layers cause over-smoothing. (3) Interpretability: hard to identify which substructures drive classification. (4) Data efficiency: GNNs need more labeled graphs than fingerprint-based methods for molecular tasks. + +--- + +**38. WL Test (Weisfeiler-Lehman)** +**Definition:** A color refinement algorithm for testing graph isomorphism, which also bounds the expressive power of message-passing GNNs. +### Theory & Explanation +The WL test determines whether two graphs are non-isomorphic. The algorithm: (1) Assign each node an initial color. (2) Iteratively refine colors: for each node, hash the multiset of its current color and its neighbors' colors to produce a new color. (3) After each iteration, compare the color histogram of both graphs. If histograms differ, graphs are non-isomorphic. The WL test fails on some regular graphs. The key connection to GNNs: Xu et al. (2018) proved that message-passing GNNs (GCN, GAT, GraphSAGE) are at most as powerful as the WL test at distinguishing graph structures. The most powerful GNN in the WL class uses SUM aggregation (not MEAN or MAX) and injective neighbor aggregation functions (GIN — Graph Isomorphism Network). +### Example +Two graphs: Graph A: (1-2, 2-3, 3-1, 3-4) — triangle + leaf. Graph B: (1-2, 2-3, 3-4, 4-1) — square (4-cycle). WL test: iteration 1 — all nodes degree 1 or 2 → colors: Graph A has one node degree 1 (leaf), three degree 2; Graph B has four degree 2. Histograms differ → non-isomorphic. +### Interview Questions +**Q: What does it mean that GNNs are "at most as powerful as the WL test"?** +A: It means the set of graph pairs that a GNN can distinguish is a subset of the pairs the WL test can distinguish. If two graphs get different WL color histograms, a sufficiently powerful GNN can distinguish them. If WL fails (same histograms for non-isomorphic graphs), no GNN can distinguish them. This is a fundamental limitation. + +**Q: Which GNN aggregation function is most expressive?** +A: SUM is the most expressive. MEAN loses information about multiset size. MAX loses both size and frequency information. GIN uses SUM + MLP to achieve WL-level expressivity. However, SUM is sensitive to graph size — for tasks where size is irrelevant, MEAN may work better in practice. + +--- + +**39. Property Graph** +**Definition:** A graph data model where both nodes and edges can have arbitrary key-value properties, used by most production graph databases. +### Theory & Explanation +The property graph model is the most widely used graph database model (Neo4j, Amazon Neptune, JanusGraph). Key characteristics: (1) Nodes represent entities and can have labels (types) and properties. (2) Edges represent relationships with a direction, a type, and optional properties. (3) A node can have multiple labels, and an edge has exactly one type. Unlike RDF, property graphs do not require URIs. The standard query language is Cypher (Neo4j): `MATCH (a:Person {name: "Alice"})-[r:KNOWS]->(b:Person) RETURN b.name, r.since`. Property graphs excel at: graph traversal algorithms, path finding, recommendation, fraud detection. +### Example +Social network as property graph: Node (Person: {name: "Alice", age: 30}), Node (Person: {name: "Bob", age: 25}), Edge (KNOWS: {since: 2020}) from Alice to Bob. Cypher: `MATCH (a:Person {name: "Alice"})-[r:KNOWS]-(friend) WHERE r.since > 2019 RETURN friend.name` → Returns "Bob". +### Interview Questions +**Q: Compare property graph model with RDF model.** +A: Property graph: simpler, more intuitive, efficient for traversal algorithms, excellent ecosystem (Neo4j, Cypher). Schema-flexible. RDF: web-native (URIs), standards-based (W3C), supports inference via RDFS/OWL, ideal for data integration across sources. RDF is better for open-world data and knowledge graphs that need to interlink with other datasets. Property graphs are better for transactional graph applications. + +**Q: What are the limitations of the property graph model?** +A: (1) No formal semantics — no inference or consistency checking. (2) No built-in data typing. (3) No standard URI scheme for entities. (4) No W3C-equivalent standard for serialization. (5) Schema evolution requires migrations. + +--- + +**40. PageRank** +**Definition:** An algorithm that ranks nodes in a graph by their importance, based on the random surfer model — the probability of arriving at each node through random walks with occasional teleportation. +### Theory & Explanation +PageRank (Page & Brin, 1998) was the original Google ranking algorithm. The core idea: a page is important if other important pages link to it. The random surfer model: a user browses by randomly clicking links. Occasionally (with probability d, the damping factor, typically 0.85), they teleport to a random page. PageRank of a page is the probability that the random surfer is on that page. Mathematically: PR(p) = (1-d)/N + d × Σ_{q ∈ in(p)} PR(q) / out_degree(q). PageRank is query-independent, democratic (each link is a vote), and robust to spam. +### Example +Web graph: A → B → C (A links to B, B links to C). Also D → A, D → B, D → C. With d=0.85: Page A gets authority from D. Page B gets from A and D. After convergence (N=4): PR ≈ [A: 0.3, B: 0.35, C: 0.25, D: 0.1] — B is most important. +### Interview Questions +**Q: How is PageRank used beyond web search?** +A: (1) Social network analysis: find influential users. (2) Recommendation: personalized PageRank for "items you might like." (3) Biological networks: rank genes by importance. (4) Knowledge graphs: identify central entities. (5) NLP: TextRank for keyword extraction and summarization. + +**Q: What are the limitations of PageRank?** +A: (1) Topic drift — a page about "AI" linked by a "sports" site dilutes relevance. (2) Freshness — old pages stay high-ranked forever. (3) Link spam — link farms can inflate PageRank. (4) Cold start — new pages have no incoming links. (5) Computation cost on billion-node graphs. + +--- + +**41. Graph-of-Thoughts (GoT)** +**Definition:** A reasoning framework where thoughts are nodes in a graph and dependencies between thoughts are edges, enabling parallel exploration, merging, and refinement. +### Theory & Explanation +GoT (Besta et al., 2023) generalizes Chain-of-Thought (linear chain) and Tree-of-Thoughts (tree with branching). In GoT, reasoning is a directed graph: nodes = individual thoughts, edges = dependencies. This enables: (1) Branching — explore multiple hypotheses in parallel. (2) Merging — combine insights from different branches. (3) Refinement — loop back and improve earlier thoughts. (4) Aggregation — combine partial results into a final answer. GoT operations: create thought, refine thought, merge thoughts, and loop. GoT outperforms CoT and ToT on tasks requiring multi-source reasoning, iterative improvement, and complex planning. +### Example +Solving a complex math problem: Node A: "identify variables" → Node B: "formulate equation 1" → Node C: "formulate equation 2" (parallel to B) → Node D: "merge equations" (combines B and C) → Node E: "solve merged system" → Node F: "verify solution" → if F fails, create Node G: "refine" (loops back). The graph structure allows backtracking without discarding work. +### Interview Questions +**Q: How does GoT compare with CoT and ToT?** +A: CoT is linear — one path. ToT is a tree — multiple parallel paths with best-first selection. GoT is a graph — paths can converge (merge insights), diverge (explore), and loop (refine). GoT is strictly more expressive than ToT, and ToT more expressive than CoT. Use CoT for simple reasoning, ToT for tasks with clear intermediate choices, GoT for complex multi-source tasks. + +**Q: What are the practical challenges of implementing GoT?** +A: (1) Graph management — maintaining a dynamic graph of thoughts. (2) Operation selection — deciding when to branch, merge, or refine requires a meta-reasoner. (3) Cost — each node requires an LLM call. (4) Evaluation — no standardized benchmarks. (5) Stopping criteria — confidence threshold, iteration limit, or convergence detection. + +--- + +## Additional Fine-Tuning Concepts + +**42. P-Tuning v2** +**Definition:** A parameter-efficient fine-tuning method that learns continuous (soft) prompt embeddings at every layer of the transformer, not just the input layer. +### Theory & Explanation +P-Tuning v2 (Liu et al., 2023) builds on prompt tuning (learn a small set of continuous prompt tokens prepended to the input). The key innovation: prompt tuning only adds learnable tokens to the input embedding layer, which limits expressivity. P-Tuning v2 adds learnable prompts to every transformer layer's hidden states. At each layer i, a learnable matrix P_i (prompt_length × hidden_dim) is inserted. This matches the expressivity of full fine-tuning while only training ~0.1-3% of parameters. P-Tuning v2 bridges the quality gap with full fine-tuning on harder tasks (NLU, sequence labeling) where standard prompt tuning lags significantly. Prompt lengths of 10-50 tokens per layer work best. +### Example +Fine-tuning a 7B model with P-Tuning v2: num_layers=32, prompt_length=20, hidden_dim=4096 → trainable params = 32 × 20 × 4096 = 2.6M (< 0.04% of 7B). This achieves 95%+ of full fine-tuning performance on GLUE benchmarks, vs standard prompt tuning at 80-85%. Training memory: ~8 GB vs ~60+ GB for full FT. +### Interview Questions +**Q: How does P-Tuning v2 differ from prefix tuning?** +A: Both add learnable tokens to each layer, but: Prefix tuning prepends learnable key-value pairs to the attention mechanism at each layer. P-Tuning v2 prepends learnable tokens to the hidden states at each layer. P-Tuning v2 recommends adding prompts to every layer and using a classification head on prompt tokens for NLU tasks. + +**Q: When is P-Tuning v2 preferred over LoRA?** +A: P-Tuning v2 is preferred when: (1) The task primarily involves modifying model behavior without changing factual knowledge. (2) You need per-layer control. (3) You want the most parameter-efficient approach (0.1% vs 0.5-2% for LoRA). LoRA is preferred when: (1) The task requires adapting to new factual knowledge. (2) You want simpler implementation. (3) You need zero inference overhead (merge adapters into base weights). + +--- + +**43. IA³ (Infused Adapter by Inhibiting and Amplifying Activations)** +**Definition:** An extremely parameter-efficient fine-tuning method that learns scaling vectors applied to key, value, and feed-forward activations, using only ~0.01% of parameters. +### Theory & Explanation +IA³ (Liu et al., 2022) learns three scaling vectors: l_k (applied to keys), l_v (applied to values), and l_ff (applied to feed-forward activations). Each vector has dimension equal to the corresponding layer's feature dimension. The scaling operation: output = s ⊙ a. Total trainable params ≈ 3 × num_layers × hidden_dim. For a 7B model: 3 × 32 × 4096 ≈ 393K parameters (< 0.006% of 7B). Despite extreme efficiency, IA³ achieves competitive performance with full fine-tuning (within 1-3% on most NLU benchmarks). Training memory overhead is negligible — can fine-tune a 7B model on a single GPU. +### Example +Fine-tuning LLaMA-7B with IA³: 393K trainable parameters. Memory: ~16 GB (half-precision) — fits on a single RTX 3090. After fine-tuning on sentiment classification: accuracy 93.5% vs full FT 94.2%. The scaling vectors can be folded into base weights, adding zero inference overhead. +### Interview Questions +**Q: How does IA³ compare with LoRA?** +A: IA³ is more parameter-efficient (0.006% vs 0.5-2%) and has lower training memory. However, IA³'s scaling vectors can only amplify or suppress existing activations, not create new feature combinations. LoRA can learn new patterns through its low-rank update. On complex tasks, LoRA typically outperforms IA³ by 1-3%. + +**Q: How do you deploy IA³ in production?** +A: IA³'s scaling vectors can be folded into base weights: W_key_new = diag(l_k) × W_key_original. Zero inference overhead. Storage is tiny (few KB per model). Ideal for: edge deployment, serving many fine-tuned variants from one base model, and rapid adapter switching. + +--- + +**44. Multi-task Fine-tuning** +**Definition:** Training a model on multiple tasks simultaneously, using a shared representation that benefits all tasks through cross-task transfer. +### Theory & Explanation +Instead of fine-tuning a separate model per task, multi-task fine-tuning trains one model on many tasks at once. Each batch contains examples from different tasks with task-specific loss functions. Shared transformer layers learn universal representations. Key challenges: (1) Task balancing — use loss weighting or proportional sampling. (2) Negative transfer — conflicting tasks degrade each other. (3) Catastrophic forgetting — interleaving tasks helps. T5 and FLAN are prominent examples — they frame every task as text-to-text, enabling unified multi-task training. +### Example +FLAN fine-tuned T5 on 60+ tasks simultaneously: classification, QA, summarization, translation. Each task is reformatted as an instruction. After multi-task training, the model generalizes to unseen tasks — it can perform tasks never explicitly trained on. FLAN outperforms single-task fine-tuning on 20 of 25 evaluation tasks. +### Interview Questions +**Q: How do you handle conflicting tasks in multi-task learning?** +A: Solutions: (1) Task-specific adapters — shared base + task-specific small modules. (2) Uncertainty weighting — weight each task's loss by model uncertainty. (3) Gradient surgery (PCGrad) — when gradients conflict, project one onto the normal of the other. (4) Progressive training — train on similar tasks first, add conflicting tasks later. + +**Q: How does multi-task fine-tuning compare to single-task + prompt engineering?** +A: Multi-task creates one model for many tasks, reducing deployment complexity and enabling zero-shot generalization. Single-task fine-tuning often achieves higher per-task performance. Prompt engineering is simplest but limited. Pattern: single-task for the most important task, multi-task for supporting tasks, prompt engineering for simple tasks. + +--- + +**45. Domain Adaptation** +**Definition:** Continuing pre-training on a domain-specific corpus to bridge the distribution gap between general pre-training data and the target domain. +### Theory & Explanation +General LLMs are pre-trained on internet text. When applied to specialized domains, there's a domain gap. Domain adaptation closes this through continued pre-training (DAPT). The process: (1) Collect domain text (1-100 GB). (2) Continue masked language modeling on this corpus. (3) The model learns domain-specific vocabulary, terminology, and conventions. (4) Then fine-tune on downstream tasks. Notable examples: BioBERT (PubMed), ClinicalBERT (clinical notes), LegalBERT (legal documents), CodeBERT (code). Domain adaptation improves in-domain performance by 2-10%. +### Example +BioBERT: BERT pre-trained on general text → continue MLM on PubMed abstracts (4.5B words) + PMC full-text (13.5B words). Result: BioBERT achieves 85% F1 on biomedical NER vs 80% for BERT-base — 5% improvement from domain adaptation alone. +### Interview Questions +**Q: How much domain data is needed?** +A: 100M-1B tokens minimum. For narrow domains, 10M-100M tokens can still help. The data must be high-quality and diverse. Diminishing returns after ~10B tokens. + +**Q: How does domain adaptation differ from full pre-training?** +A: Full pre-training starts from random weights on trillions of tokens (months, thousands of GPUs). Domain adaptation continues from a pre-trained model on billions of tokens (days, single GPU). Much cheaper — BioBERT trained on 1 GPU for 10 days vs millions of GPU-hours for BERT. + +--- + +**46. Parameter Efficiency Comparison Table** +**Definition:** A structured comparison of PEFT methods across key dimensions. + +| Method | Trainable Params % | Training Memory (7B) | Quality vs Full FT | Inference Overhead | +|---|---|---|---|---| +| **Full Fine-Tuning** | 100% | ~120 GB (FP16) | Baseline | None | +| **LoRA (r=8)** | 0.5-2% | ~32 GB | 95-99% | None* | +| **QLoRA** | 0.5-2% | ~10 GB (4-bit) | 94-98% | None* | +| **Adapter** | 3-6% | ~40 GB | 97-99% | Small | +| **Prompt Tuning** | 0.01-0.1% | ~16 GB | 80-90% | Small | +| **Prefix Tuning** | 0.1-1% | ~20 GB | 85-95% | Small | +| **P-Tuning v2** | 0.1-3% | ~24 GB | 93-98% | Medium | +| **IA³** | 0.006% | ~16 GB | 90-95% | None* | + +* Zero overhead when merged into base weights. + +### Interview Questions +**Q: How do you choose the right PEFT method?** +A: Single GPU (24 GB or less) → QLoRA or IA³. Need zero inference overhead → LoRA or IA³. Quality paramount → full FT or LoRA (r=64-128). Many task-specific adapters → IA³ (few KB each). NLU behavior modification → P-Tuning v2. Multi-task → LoRA with task-specific adapters. + +**Q: What are the practical implications of QLoRA's 4-bit quantization?** +A: Reduces memory 4x: LLaMA-65B fits on a single 48GB GPU. Quality drop only 1-2% vs full FT. Benefits: fine-tune 7B models on consumer GPUs, 65B+ on single A100. Trade-off: training 20-30% slower due to quantization/dequantization overhead. + +--- + +## Additional Evaluation & Hallucination Concepts + +**47. BLEURT** +**Definition:** A learned evaluation metric based on BERT, trained on human judgments to measure text generation quality with higher correlation to human evaluation than traditional metrics. +### Theory & Explanation +BLEURT (Sellam et al., 2020) addresses the limitation of n-gram metrics (BLEU, ROUGE): they don't capture semantic similarity. BLEURT fine-tunes BERT on two stages: (1) Pre-training on synthetic data: generate millions of (reference, candidate) pairs with controlled perturbations (masking, back-translation, word dropout) and train BERT to predict the perturbation type. (2) Fine-tuning on human judgments: fine-tune on WMT human evaluation data. The final model takes (reference, candidate) as input and outputs a quality score. BLEURT achieves 0.45-0.55 Pearson correlation with human judgments on translation, vs 0.20-0.35 for BLEU. +### Example +Reference: "The cat sat on the mat." Candidate A: "The feline rested on the rug." Candidate B: "Cat mat sat the." BLEU: A = 0.3, B = 0.25. BLEURT: A = 0.85, B = 0.15. BLEURT correctly identifies A as better because it captures semantic equivalence. +### Interview Questions +**Q: How does BLEURT compare with BERTScore?** +A: BERTScore computes pairwise cosine similarity between reference and candidate token embeddings. It requires no training. BLEURT fine-tunes BERT end-to-end on human judgment data. BLEURT correlates better with human judgment but is more expensive and needs domain-specific training data. + +**Q: What are the limitations of BLEURT?** +A: (1) Reference-dependent — can't evaluate open-ended generation. (2) Domain sensitivity — trained on translation data may not work for summarization. (3) Computational cost — full BERT forward pass per pair. (4) Score interpretation — only relative comparisons are valid. + +--- + +**48. Factual Consistency** +**Definition:** Measuring whether a generated text is factually consistent with its source — the opposite of hallucination. +### Theory & Explanation +Factual consistency checks if the generation adds, contradicts, or distorts information from the source. The dominant approach uses NLI: treat source as premise, generation as hypothesis, classify as entailment (consistent) or contradiction (inconsistent). Key systems: FactCC, SummaC (sentence-level NLI with max contradiction aggregation), AlignScore (unified alignment model), SelfCheckGPT (multiple generations, no source needed). NLI-based approaches achieve 70-90% accuracy. Challenges: correct but not in source (general knowledge), verbose generations (decompose into atomic claims). +### Example +Source: "The Eiffel Tower is 330 meters tall and was completed in 1889." Generation: "The Eiffel Tower, a 330-meter landmark, was finished in 1889. It was designed by Gustave Eiffel." Sentence 1: entailment. Sentence 2: "designed by Eiffel" — not in source but well-known fact → neutral (decision depends on strict vs permissive mode). +### Interview Questions +**Q: How do you handle the strict vs permissive consistency trade-off?** +A: Strict mode: only accept information in the source (best for summarization). Permissive mode: accept verifiably true information (best for QA). Hybrid: if a claim is not in source, check against a knowledge base or LLM. Use strict for high-stakes domains (medical, legal). + +**Q: Compare NLI-based vs LLM-based factual consistency.** +A: NLI-based: faster, deterministic, interpretable, but limited to single-source comparison. LLM-based: more flexible, can use general knowledge, provides explanations, but slower and less deterministic. Best practice: use NLI for high-volume automated evaluation, LLM for detailed analysis of edge cases. + +--- + +**49. A/B Evaluation** +**Definition:** A live evaluation method where two systems each serve a portion of user traffic, and performance is compared on real user engagement metrics. +### Theory & Explanation +A/B testing evaluates systems with real users in production. Process: (1) Define hypothesis. (2) Split traffic randomly (50/50). (3) Run for sufficient duration (1-2 weeks). (4) Measure metrics: primary (task success rate, satisfaction) and secondary (latency, cost). (5) Statistical analysis (p < 0.05). (6) Decision. Challenges: novelty effect, network effects, multiple metric testing (Bonferroni correction). +### Example +RAG chatbot A (no citations) vs B (with citations). Metrics: Task success: A=72%, B=78% (p=0.03). User satisfaction: A=3.8, B=4.1 (p=0.01). Conclusion: B improves success and satisfaction. Roll out to all users. +### Interview Questions +**Q: What are the risks of A/B testing in LLM applications?** +A: (1) Quality risk — bad system harms user trust. Start with small traffic (5-10%). (2) Cost risk — expensive model improves metrics but costs 10x. (3) Randomization bias — consistent cookie-based assignment. (4) Content safety — moderation in the loop. (5) Interaction effects — segment results by query type. + +**Q: How long should an A/B test run?** +A: 2 weeks minimum for high-traffic systems (100K+ queries/day), 4 weeks for low-traffic. Consider weekly patterns. Stop early only if clearly harmful or overwhelmingly significant. + +--- + +**50. ARES (Automated RAG Evaluation System)** +**Definition:** An LLM-based evaluation framework specifically designed for RAG systems, using three fine-tuned evaluator models for context relevance, answer faithfulness, and answer relevance. +### Theory & Explanation +ARES (Saad-Falcon et al., 2023) works in three stages: (1) Synthetic data generation: use an LLM to generate synthetic questions, contexts, and answers from the knowledge base. (2) Train evaluators: fine-tune three lightweight LLM classifiers: Context Relevance evaluator, Answer Faithfulness evaluator, Answer Relevance evaluator. (3) Evaluate: run all three evaluators and aggregate scores. ARES achieves 85-95% agreement with human evaluators. The key advantage: evaluators are domain-adapted. +### Example +RAG system for legal documents. ARES generates: Question: "What is the statute of limitations for breach of contract in California?" Retrieved context: legal document chunk. Answer: [system answer]. Evaluators score: context relevance (0.9), answer faithfulness (0.85), answer relevance (0.88). Combined score = 0.88. +### Interview Questions +**Q: How does ARES differ from using a general LLM-as-judge?** +A: ARES trains domain-specific evaluators on synthetic data from your corpus, making them more accurate for your specific domain. General LLM-as-judge is prompt-dependent and may not capture domain-specific notions of relevance and faithfulness. ARES is more expensive to set up (requires training) but provides more consistent, interpretable evaluations. + +**Q: What are the failure modes of ARES?** +A: (1) Synthetic data quality — if generated data doesn't reflect real queries, evaluators won't generalize. (2) Evaluator model capacity — lightweight classifiers may miss nuanced patterns. (3) Domain shift — if the knowledge base changes significantly, evaluators need retraining. (4) Three scores don't capture all quality dimensions (e.g., coherence, safety). + +--- + +**51. TruLens** +**Definition:** An evaluation toolkit for LLM applications that measures groundedness, relevance, and coherence using feedback functions. +### Theory & Explanation +TruLens provides three core evaluation dimensions: (1) Groundedness: is the output supported by the retrieved context? Uses NLI to check each sentence against the context. (2) Relevance: does the output address the input question/query? Measures semantic similarity between input and output. (3) Coherence: is the output well-structured and logically flowing? Measures readability and structure. Each dimension is scored 0-1 via feedback functions. TruLens also provides: dashboard for tracking evaluations over time, app comparison (A/B testing), and guardrails (threshold-based alerts). It integrates with LangChain, LlamaIndex, and custom applications. The feedback functions can use: NLI models, embedding similarity, LLM-as-judge, or custom logic. +### Interview Questions +**Q: How do TruLens' three dimensions map to RAG quality?** +A: Groundedness → retrieval quality (did the system use the context correctly?). Relevance → query understanding (did the system answer the right question?). Coherence → generation quality (is the output well-formed?). Together they provide a holistic view: a system could have high relevance and coherence but low groundedness (fluent but hallucinated), or high groundedness and relevance but low coherence (correct but poorly structured). + +**Q: How would you use TruLens in a CI/CD pipeline?** +A: Run TruLens evaluation on a test set after each model update or RAG pipeline change. Set thresholds (e.g., groundedness > 0.8, relevance > 0.85). If any score drops below threshold, block deployment. Track scores over time to detect regression. Use the dashboard to compare candidate and production systems side by side. + +--- + +**52. Prompt-based Hallucination Detection** +**Definition:** Using an LLM to determine whether a generated response contradicts or is unsupported by its source context, through carefully crafted detection prompts. +### Theory & Explanation +The simplest hallucination detection approach: ask an LLM evaluator directly. For example: "Does the following answer contain information not supported by the context? Answer only 'Yes' or 'No'." This can be enhanced with chain-of-thought: "List each factual claim in the answer. For each claim, determine if it's supported by the context. Then provide your verdict." Key design choices: (1) Granularity: sentence-level, claim-level, or holistic. (2) Prompt specificity: binary classification vs detailed analysis. (3) Model selection: same model (faster, may have same blind spots) or stronger model (more accurate, slower, costlier). Accuracy is 70-90% depending on task complexity and model strength. Best combined with other methods for production use. +### Example +Prompt: "Context: {retrieved_chunks}\n\nAnswer: {generated_answer}\n\nIdentify each factual claim in the answer. For each claim, mark it as: Supported (directly stated in context), Contradicted (context says the opposite), or Not in Context (not mentioned in context). Then give an overall verdict: Consistent, Partially Consistent, or Inconsistent." +### Interview Questions +**Q: What are the failure modes of prompt-based detection?** +A: (1) Same-model bias — if the generator and evaluator are the same model, it may confirm its own hallucinations. Mitigation: use a different, stronger model for evaluation. (2) Prompt sensitivity — small prompt changes produce different results. Mitigation: use structured prompts with clear output format. (3) Overconfidence — the evaluator may confidently misclassify. Mitigation: request confidence scores alongside verdicts. (4) Long context — with many claims, the evaluator may miss some. Mitigation: decompose into atomic claims first. + +**Q: When is prompt-based detection preferable to NLI-based?** +A: Prompt-based is preferable when: (1) You need flexible, open-ended evaluation (not just entailment/contradiction). (2) The source context is multi-document or conversational (NLI assumes single premise). (3) You want explanations alongside scores. (4) You're already using an LLM for generation and can reuse it. NLI-based is better for: high-throughput, low-latency settings, and when you need consistent, deterministic outputs. + +--- + +**53. Token-level Probability Detection** +**Definition:** Detecting hallucinations by analyzing token-level probabilities — low probability on key factual tokens signals potential hallucination. +### Theory & Explanation +The intuition: when a model hallucinates, it tends to generate key information tokens (dates, names, numbers) with lower probability than when it generates grounded facts. The approach: (1) Identify information tokens in the generated text (entities, dates, numbers, technical terms) using NER or regex. (2) Extract the model's log probability for each information token at generation time. (3) Aggregate: low average probability → high hallucination risk. Simple metrics: mean probability of information tokens, minimum probability, or entropy across the sequence. More sophisticated: semantic entropy (cluster multiple generations by meaning, compute entropy across clusters). Token-level methods are fast (no additional LLM calls) but less accurate than consistency-based methods (70-80% accuracy). They work best for detecting simple factual errors (wrong date, wrong name) but miss contextual inconsistency (contradicting earlier statements, violating common sense). +### Example +Generation: "The Eiffel Tower was completed in 1889 and is 330 meters tall." Token probabilities: the=0.99, Eiffel=0.97, Tower=0.98, was=0.99, completed=0.95, in=0.99, 1889=0.45 (low — model uncertain about the year), and=0.99, is=0.99, 330=0.52 (low — model uncertain about height), meters=0.97, tall=0.99. Average on information tokens (1889, 330): 0.485 → below threshold → flag as potential hallucination. Correct: the model was indeed uncertain about the exact numbers. +### Interview Questions +**Q: What are the advantages and disadvantages of token-level detection?** +A: Advantages: (1) Fast — no additional LLM calls, uses generation-time probabilities. (2) Cheap — minimal computational overhead. (3) Real-time — can be applied during generation to trigger retrieval (FLARE) or regeneration. Disadvantages: (1) Lower accuracy (70-80%) compared to consistency-based methods (85-95%). (2) Only catches uncertainty, not confident hallucinations (the model can be confidently wrong with high token probabilities). (3) Doesn't detect contextual inconsistencies (contradicting information spread across sentences). (4) Requires access to token-level log probabilities (not all APIs expose these). + +**Q: How does semantic entropy improve on simple token-level probability?** +A: Token-level probability captures per-token uncertainty but misses semantic consistency — the model might be confident on each token but the overall meaning is wrong. Semantic entropy: generate multiple responses (N=5-10 with temperature > 0), cluster them by semantic meaning (using entailment or embedding similarity), compute entropy across clusters. High semantic entropy = many different meanings across generations = high hallucination risk. This captures both token-level uncertainty and meaning-level inconsistency. Cost: N times more generation calls. diff --git a/techy/concept-reference.md b/techy/concept-reference.md new file mode 100644 index 0000000..39a93de --- /dev/null +++ b/techy/concept-reference.md @@ -0,0 +1,473 @@ +# Complete Concept Reference — Master Index + +> Every concept listed below links to its explanation in the study guide files. +> File anchors use GitHub-style markdown heading anchors (lowercase, hyphens for spaces, punctuation removed). + +--- + +## 1. PYTHON (for Data Science) + +| Concept | Covered In | File | +|---------|-----------|------| +| List, Tuple, Set, Dict | Built-in Python knowledge — referenced across all files | General | +| List Comprehension, Lambda, map/filter/reduce | Used in code examples throughout | All files | +| NumPy array, Pandas Series/DataFrame | Data manipulation context in [01-data-science.md](./01-data-science.md#example-10) | [01](./01-data-science.md) | +| Missing data (NaN), dropna(), fillna() | Data cleaning context in [01-data-science.md](./01-data-science.md#eda-exploratory-data-analysis) | [01](./01-data-science.md#eda-exploratory-data-analysis) | +| Generator, Decorator, `is` vs `==` | Python-specific gotchas — implicit knowledge | General | +| `*args`/`**kwargs`, mutable default args | Python-specific — not directly covered | General | + +--- + +## 2. DATA SCIENCE (General) + +### CRISP-DM +- **Status:** ❌ NOT COVERED — needs to be added +- **Target file:** [01-data-science.md](./01-data-science.md) + +### Supervised, Unsupervised, Semi-supervised, RL +| Concept | Section | File | +|---------|---------|------| +| Supervised Learning | [Semi-supervised Learning](./01-data-science.md#semi-supervised-learning) — Theory & Explanation | [01](./01-data-science.md#semi-supervised-learning) | +| Unsupervised Learning | K-Means, PCA, clustering methods in [02-machine-learning.md](./02-machine-learning.md#k-means-clustering) | [02](./02-machine-learning.md#k-means-clustering) | +| Semi-supervised Learning | [Semi-supervised Learning](./01-data-science.md#semi-supervised-learning) — full section | [01](./01-data-science.md#semi-supervised-learning) | +| Reinforcement Learning | [Reinforcement Learning](./01-data-science.md#reinforcement-learning) — full section | [01](./01-data-science.md#reinforcement-learning) | + +### Overfitting / Underfitting / Bias-Variance Tradeoff +- **Status:** ❌ NOT COVERED as standalone sections +- Overfitting mentioned in context of [Decision Trees](./02-machine-learning.md#decision-trees), [XGBoost](./02-machine-learning.md#xgboost), [Random Forest](./02-machine-learning.md#random-forest) and [Autoencoder](./03-deep-learning-and-nlp.md#autoencoder) +- **Target file:** [02-machine-learning.md](./02-machine-learning.md) + +### Cross-Validation / Stratified CV +- **Status:** ❌ NOT COVERED as standalone sections +- Cross-validation referenced in [Hyperparameter Tuning](./02-machine-learning.md#hyperparameter-tuning) example +- **Target file:** [02-machine-learning.md](./02-machine-learning.md) + +### Feature Engineering / Feature Selection +| Concept | Section | File | +|---------|---------|------| +| Feature Engineering | Referenced in EDA section [01-data-science.md](./01-data-science.md#eda-exploratory-data-analysis) | [01](./01-data-science.md#eda-exploratory-data-analysis) | +| Feature Selection | PCA as dim reduction [02-machine-learning.md](./02-machine-learning.md#pca-principal-component-analysis) | [02](./02-machine-learning.md#pca-principal-component-analysis) | + +### Imbalanced Data +| Concept | Section | File | +|---------|---------|------| +| Imbalanced Data | [Imbalanced Data](./01-data-science.md#imbalanced-data) — full section | [01](./01-data-science.md#imbalanced-data) | +| SMOTE, Class Weights | [Imbalanced Data](./01-data-science.md#example-8) — example | [01](./01-data-science.md#example-8) | + +### EDA (Exploratory Data Analysis) +| Concept | Section | File | +|---------|---------|------| +| EDA | [EDA](./01-data-science.md#eda-exploratory-data-analysis) — full section | [01](./01-data-science.md#eda-exploratory-data-analysis) | + +### A/B Testing +| Concept | Section | File | +|---------|---------|------| +| A/B Testing | [A/B Testing](./01-data-science.md#ab-testing) — full section | [01](./01-data-science.md#ab-testing) | +| p-value | [A/B Testing](./01-data-science.md#theory--explanation-31) — Theory & Explanation | [01](./01-data-science.md#theory--explanation-31) | +| Shapiro-Wilk Test | [Shapiro-Wilk Test](./01-data-science.md#shapiro-wilk-test) — full section | [01](./01-data-science.md#shapiro-wilk-test) | +| Levene's Test | [Levene's Test](./01-data-science.md#levenes-test) — full section | [01](./01-data-science.md#levenes-test) | + +### Pipelines (sklearn) +- **Status:** ❌ NOT COVERED — needs to be added +- **Target file:** [02-machine-learning.md](./02-machine-learning.md) + +### SHAP / LIME +| Concept | Section | File | +|---------|---------|------| +| SHAP | Referenced in [EDA](./01-data-science.md#example-13) example | [01](./01-data-science.md#example-13) | +| LIME | Referenced in [EDA](./01-data-science.md#example-13) example | [01](./01-data-science.md#example-13) | + +### Data Science Concepts (Infrastructure) +| Concept | Section | File | +|---------|---------|------| +| Data Engineering vs Data Science | [Data Engineering vs Data Science](./01-data-science.md#data-engineering-vs-data-science) | [01](./01-data-science.md#data-engineering-vs-data-science) | +| Data Warehousing | [Data Warehousing](./01-data-science.md#data-warehousing) | [01](./01-data-science.md#data-warehousing) | +| ETL vs ELT | [ETL vs ELT](./01-data-science.md#etl-vs-elt) | [01](./01-data-science.md#etl-vs-elt) | +| Data Modeling (Star/Snowflake) | [Data Modeling](./01-data-science.md#data-modeling-star-schema-snowflake-schema) | [01](./01-data-science.md#data-modeling-star-schema-snowflake-schema) | +| Data Lake | [Data Lake](./01-data-science.md#data-lake) | [01](./01-data-science.md#data-lake) | +| Data Pipeline | [Data Pipeline](./01-data-science.md#data-pipeline) | [01](./01-data-science.md#data-pipeline) | +| Data Governance | [Data Governance](./01-data-science.md#data-governance) | [01](./01-data-science.md#data-governance) | +| Data Catalog | [Data Catalog](./01-data-science.md#data-catalog) | [01](./01-data-science.md#data-catalog) | +| Data Lineage | [Data Lineage](./01-data-science.md#data-lineage) | [01](./01-data-science.md#data-lineage) | +| Data Quality | [Data Quality](./01-data-science.md#data-quality) | [01](./01-data-science.md#data-quality) | +| Data Versioning | [Data Versioning](./01-data-science.md#data-versioning) | [01](./01-data-science.md#data-versioning) | +| Feature Store | [Feature Store](./01-data-science.md#feature-store) | [01](./01-data-science.md#feature-store) | +| Bias (Data) | [Bias (Data)](./01-data-science.md#bias-data) — full section | [01](./01-data-science.md#bias-data) | +| Data Science Workflow | [Data Science Workflow](./01-data-science.md#data-science-workflow) — full section | [01](./01-data-science.md#data-science-workflow) | + +--- + +## 3. MACHINE LEARNING + +### Regression Algorithms +| Concept | Section | File | +|---------|---------|------| +| Linear Regression | [Linear Regression](./02-machine-learning.md#linear-regression) — full section | [02](./02-machine-learning.md#linear-regression) | +| Logistic Regression | [Logistic Regression](./02-machine-learning.md#logistic-regression) — full section | [02](./02-machine-learning.md#logistic-regression) | + +### Tree-Based & Ensemble +| Concept | Section | File | +|---------|---------|------| +| Decision Trees | [Decision Trees](./02-machine-learning.md#decision-trees) — full section | [02](./02-machine-learning.md#decision-trees) | +| Random Forest | [Random Forest](./02-machine-learning.md#random-forest) — full section | [02](./02-machine-learning.md#random-forest) | +| XGBoost | [XGBoost](./02-machine-learning.md#xgboost) — full section | [02](./02-machine-learning.md#xgboost) | +| LightGBM | [LightGBM](./02-machine-learning.md#lightgbm) — full section | [02](./02-machine-learning.md#lightgbm) | +| Gradient Boosting (General) | [Gradient Boosting](./02-machine-learning.md#gradient-boosting-general) — full section | [02](./02-machine-learning.md#gradient-boosting-general) | +| Bagging | Referenced in Random Forest section | [02](./02-machine-learning.md#random-forest) | +| Boosting | Referenced in Gradient Boosting section | [02](./02-machine-learning.md#gradient-boosting-general) | +| Stacking | [Stacking](./02-machine-learning.md#stacking) — full section | [02](./02-machine-learning.md#stacking) | + +### Other Algorithms +| Concept | Section | File | +|---------|---------|------| +| K-Nearest Neighbors | [K-Nearest Neighbors](./02-machine-learning.md#k-nearest-neighbors) — full section | [02](./02-machine-learning.md#k-nearest-neighbors) | +| SVM | [SVM](./02-machine-learning.md#support-vector-machine-svm) — full section | [02](./02-machine-learning.md#support-vector-machine-svm) | +| Naive Bayes | [Naive Bayes](./02-machine-learning.md#naive-bayes) — full section | [02](./02-machine-learning.md#naive-bayes) | + +### Clustering +| Concept | Section | File | +|---------|---------|------| +| K-Means | [K-Means](./02-machine-learning.md#k-means-clustering) — full section | [02](./02-machine-learning.md#k-means-clustering) | +| Hierarchical Clustering | [Hierarchical Clustering](./02-machine-learning.md#hierarchical-clustering) | [02](./02-machine-learning.md#hierarchical-clustering) | +| DBSCAN | [DBSCAN](./02-machine-learning.md#dbscan) | [02](./02-machine-learning.md#dbscan) | + +### Dimensionality Reduction +| Concept | Section | File | +|---------|---------|------| +| PCA | [PCA](./02-machine-learning.md#pca-principal-component-analysis) — full section | [02](./02-machine-learning.md#pca-principal-component-analysis) | +| t-SNE | [t-SNE](./02-machine-learning.md#t-sne) — full section | [02](./02-machine-learning.md#t-sne) | + +### Anomaly Detection +| Concept | Section | File | +|---------|---------|------| +| Isolation Forest | [Isolation Forest](./02-machine-learning.md#isolation-forest) — full section | [02](./02-machine-learning.md#isolation-forest) | + +### Regularization +| Concept | Section | File | +|---------|---------|------| +| L1 (Lasso) | Referenced in [XGBoost](./02-machine-learning.md#xgboost) and [Gradient Boosting](./02-machine-learning.md#gradient-boosting-general) | [02](./02-machine-learning.md#xgboost) | +| L2 (Ridge) | Referenced in [XGBoost](./02-machine-learning.md#xgboost) and [Gradient Boosting](./02-machine-learning.md#gradient-boosting-general) | [02](./02-machine-learning.md#xgboost) | +| Elastic Net | Referenced in context | [02](./02-machine-learning.md) | +| Gini Impurity | [Decision Trees](./02-machine-learning.md#decision-trees) — Theory & Explanation | [02](./02-machine-learning.md#decision-trees) | +| Entropy | [Decision Trees](./02-machine-learning.md#decision-trees) — Theory & Explanation | [02](./02-machine-learning.md#decision-trees) | + +### Early Stopping +| Concept | Section | File | +|---------|---------|------| +| Early Stopping | Covered in [XGBoost](./02-machine-learning.md#xgboost), [Gradient Boosting](./02-machine-learning.md#gradient-boosting-general) interview questions | [02](./02-machine-learning.md#xgboost) | + +### Hyperparameter Tuning +| Concept | Section | File | +|---------|---------|------| +| Grid Search, Random, Bayesian | [Hyperparameter Tuning](./02-machine-learning.md#hyperparameter-tuning) — full section | [02](./02-machine-learning.md#hyperparameter-tuning) | +| Learning Curve | [Learning Curve](./02-machine-learning.md#learning-curve) — full section | [02](./02-machine-learning.md#learning-curve) | + +### Evaluation Metrics +| Concept | Section | File | +|---------|---------|------| +| Confusion Matrix | Referenced in [Evaluation Metrics](./02-machine-learning.md#evaluation-metrics-classification--regression) Example | [02](./02-machine-learning.md#evaluation-metrics-classification--regression) | +| Accuracy, Precision, Recall, F1 | [Evaluation Metrics](./02-machine-learning.md#evaluation-metrics-classification--regression) — full section | [02](./02-machine-learning.md#evaluation-metrics-classification--regression) | +| ROC Curve / AUC | [Evaluation Metrics](./02-machine-learning.md#evaluation-metrics-classification--regression) — full section | [02](./02-machine-learning.md#evaluation-metrics-classification--regression) | +| PR Curve / AUC-PR | [Evaluation Metrics](./02-machine-learning.md#evaluation-metrics-classification--regression) — full section | [02](./02-machine-learning.md#evaluation-metrics-classification--regression) | +| MSE, RMSE, MAE | [Evaluation Metrics](./02-machine-learning.md#evaluation-metrics-classification--regression) — Example with loan default | [02](./02-machine-learning.md#evaluation-metrics-classification--regression) | +| R², Adjusted R² | [Linear Regression](./02-machine-learning.md#linear-regression) — assumptions discussion | [02](./02-machine-learning.md#linear-regression) | +| Log Loss | [Log Loss](./02-machine-learning.md#log-loss) — full section | [02](./02-machine-learning.md#log-loss) | + +--- + +## 4. DEEP LEARNING + +### Core Concepts +| Concept | Section | File | +|---------|---------|------| +| Neuron, Layer, Feedforward | [Neural Networks](./03-deep-learning-and-nlp.md#neural-networks-neuron-layer-feedforward) | [03](./03-deep-learning-and-nlp.md#neural-networks-neuron-layer-feedforward) | +| Activation Functions | [Activation Functions](./03-deep-learning-and-nlp.md#activation-functions) — full section | [03](./03-deep-learning-and-nlp.md#activation-functions) | +| ReLU | [Activation Functions](./03-deep-learning-and-nlp.md#activation-functions) — Theory | [03](./03-deep-learning-and-nlp.md#activation-functions) | +| Leaky ReLU | [Activation Functions](./03-deep-learning-and-nlp.md#activation-functions) — Theory | [03](./03-deep-learning-and-nlp.md#activation-functions) | +| Sigmoid, Tanh, Softmax | [Activation Functions](./03-deep-learning-and-nlp.md#activation-functions) — full section | [03](./03-deep-learning-and-nlp.md#activation-functions) | +| Backpropagation | [Backpropagation](./03-deep-learning-and-nlp.md#backpropagation) — full section | [03](./03-deep-learning-and-nlp.md#backpropagation) | +| Loss Function | Cross-entropy, MSE — referenced in backpropagation | [03](./03-deep-learning-and-nlp.md#backpropagation) | +| Optimizer (SGD, Adam, RMSProp) | [Optimizer](./03-deep-learning-and-nlp.md#optimizer) — full section | [03](./03-deep-learning-and-nlp.md#optimizer) | +| Learning Rate | [Learning Rate](./03-deep-learning-and-nlp.md#learning-rate) — full section | [03](./03-deep-learning-and-nlp.md#learning-rate) | +| Learning Rate Scheduler | [Learning Rate Scheduler](./03-deep-learning-and-nlp.md#learning-rate-scheduler) | [03](./03-deep-learning-and-nlp.md#learning-rate-scheduler) | +| Epoch & Batch | [Epoch & Batch](./03-deep-learning-and-nlp.md#epoch--batch) — full section | [03](./03-deep-learning-and-nlp.md#epoch--batch) | +| Gradient Descent Variants | [Gradient Descent Variants](./03-deep-learning-and-nlp.md#gradient-descent-variants) | [03](./03-deep-learning-and-nlp.md#gradient-descent-variants) | +| Vanishing / Exploding Gradient | Referenced in [Backpropagation](./03-deep-learning-and-nlp.md#backpropagation) interview | [03](./03-deep-learning-and-nlp.md#backpropagation) | +| Dropout | [Dropout](./03-deep-learning-and-nlp.md#dropout) — full section | [03](./03-deep-learning-and-nlp.md#dropout) | +| Batch Normalization | [Batch Normalization](./03-deep-learning-and-nlp.md#batch-normalization) | [03](./03-deep-learning-and-nlp.md#batch-normalization) | +| Layer Normalization | [Layer Normalization](./03-deep-learning-and-nlp.md#layer-normalization) | [03](./03-deep-learning-and-nlp.md#layer-normalization) | +| Weight Initialization (Xavier/He) | [Weight Initialization](./03-deep-learning-and-nlp.md#weight-initialization) | [03](./03-deep-learning-and-nlp.md#weight-initialization) | +| Data Augmentation | [Data Augmentation](./03-deep-learning-and-nlp.md#data-augmentation) | [03](./03-deep-learning-and-nlp.md#data-augmentation) | +| Transfer Learning | [Transfer Learning](./03-deep-learning-and-nlp.md#transfer-learning) — full section | [03](./03-deep-learning-and-nlp.md#transfer-learning) | + +### Architectures +| Concept | Section | File | +|---------|---------|------| +| CNN | [CNN](./03-deep-learning-and-nlp.md#cnn-convolutional-neural-network) — full section | [03](./03-deep-learning-and-nlp.md#cnn-convolutional-neural-network) | +| RNN | [RNN & LSTM](./03-deep-learning-and-nlp.md#rnn--lstm) — full section | [03](./03-deep-learning-and-nlp.md#rnn--lstm) | +| LSTM | [RNN & LSTM](./03-deep-learning-and-nlp.md#rnn--lstm) — full section | [03](./03-deep-learning-and-nlp.md#rnn--lstm) | +| GRU | [Sequence Models](./03-deep-learning-and-nlp.md#sequence-models-rnnlstmgruattentiontransformer) | [03](./03-deep-learning-and-nlp.md#sequence-models-rnnlstmgruattentiontransformer) | +| Bidirectional RNN | [Sequence Models](./03-deep-learning-and-nlp.md#sequence-models-rnnlstmgruattentiontransformer) | [03](./03-deep-learning-and-nlp.md#sequence-models-rnnlstmgruattentiontransformer) | +| Encoder-Decoder | [Machine Translation](./03-deep-learning-and-nlp.md#machine-translation) — Seq2Seq context | [03](./03-deep-learning-and-nlp.md#machine-translation) | +| Attention | [Transformers](./03-deep-learning-and-nlp.md#transformers) — full section | [03](./03-deep-learning-and-nlp.md#transformers) | +| Transformer | [Transformers](./03-deep-learning-and-nlp.md#transformers) — full section | [03](./03-deep-learning-and-nlp.md#transformers) | +| Autoencoder | [Autoencoder](./03-deep-learning-and-nlp.md#autoencoder) — full section | [03](./03-deep-learning-and-nlp.md#autoencoder) | +| GAN | [GAN](./03-deep-learning-and-nlp.md#gan-generative-adversarial-network) — full section | [03](./03-deep-learning-and-nlp.md#gan-generative-adversarial-network) | +| ResNet | [ResNet](./03-deep-learning-and-nlp.md#resnet) — full section | [03](./03-deep-learning-and-nlp.md#resnet) | +| U-Net | [U-Net](./03-deep-learning-and-nlp.md#u-net) — full section | [03](./03-deep-learning-and-nlp.md#u-net) | + +--- + +## 5. NATURAL LANGUAGE PROCESSING + +### Preprocessing +| Concept | Section | File | +|---------|---------|------| +| Tokenization | [Tokenization](./03-deep-learning-and-nlp.md#tokenization) — full section | [03](./03-deep-learning-and-nlp.md#tokenization) | +| BPE | [Tokenization](./03-deep-learning-and-nlp.md#tokenization) — Theory | [03](./03-deep-learning-and-nlp.md#tokenization) | +| WordPiece | [Tokenization](./03-deep-learning-and-nlp.md#tokenization) — Theory | [03](./03-deep-learning-and-nlp.md#tokenization) | +| SentencePiece | [Tokenization](./03-deep-learning-and-nlp.md#tokenization) — Theory | [03](./03-deep-learning-and-nlp.md#tokenization) | +| Stop Words, Stemming, Lemmatization | Referenced in Text Preprocessing context | [03](./03-deep-learning-and-nlp.md) | +| Bag of Words / TF-IDF | [Bag of Words & TF-IDF](./03-deep-learning-and-nlp.md#bag-of-words--tf-idf) | [03](./03-deep-learning-and-nlp.md#bag-of-words--tf-idf) | +| N-grams | [Bag of Words & TF-IDF](./03-deep-learning-and-nlp.md#bag-of-words--tf-idf) — Theory | [03](./03-deep-learning-and-nlp.md#bag-of-words--tf-idf) | + +### Word & Sentence Embeddings +| Concept | Section | File | +|---------|---------|------| +| Word Embeddings | [Word Embeddings](./03-deep-learning-and-nlp.md#word-embeddings) — full section | [03](./03-deep-learning-and-nlp.md#word-embeddings) | +| Word2Vec (CBOW/Skip-gram) | [Word Embeddings](./03-deep-learning-and-nlp.md#word-embeddings) — full section | [03](./03-deep-learning-and-nlp.md#word-embeddings) | +| GloVe | [Word Embeddings](./03-deep-learning-and-nlp.md#word-embeddings) — full section | [03](./03-deep-learning-and-nlp.md#word-embeddings) | +| FastText | [Word Embeddings](./03-deep-learning-and-nlp.md#word-embeddings) — full section | [03](./03-deep-learning-and-nlp.md#word-embeddings) | +| Sentence Embeddings | [Sentence Embeddings](./03-deep-learning-and-nlp.md#sentence-embeddings) — full section | [03](./03-deep-learning-and-nlp.md#sentence-embeddings) | +| Cosine Similarity | [Cosine Similarity](./03-deep-learning-and-nlp.md#cosine-similarity) — full section | [03](./03-deep-learning-and-nlp.md#cosine-similarity) | +| Bi-encoder vs Cross-encoder | [Bi-encoder vs Cross-encoder](./03-deep-learning-and-nlp.md#bi-encoder-vs-cross-encoder) | [03](./03-deep-learning-and-nlp.md#bi-encoder-vs-cross-encoder) | + +### Pre-trained Language Models +| Concept | Section | File | +|---------|---------|------| +| BERT | [BERT & Pre-trained Language Models](./03-deep-learning-and-nlp.md#bert--pre-trained-language-models) | [03](./03-deep-learning-and-nlp.md#bert--pre-trained-language-models) | +| RoBERTa | [Pre-trained Language Models Comparison](./03-deep-learning-and-nlp.md#pre-trained-language-models-comparison) | [03](./03-deep-learning-and-nlp.md#pre-trained-language-models-comparison) | +| DistilBERT, ALBERT | [ALBERT](./03-deep-learning-and-nlp.md#albert) — full section | [03](./03-deep-learning-and-nlp.md#albert) | +| GPT | [Pre-trained Language Models Comparison](./03-deep-learning-and-nlp.md#pre-trained-language-models-comparison) | [03](./03-deep-learning-and-nlp.md#pre-trained-language-models-comparison) | +| T5, BART, XLNet | [Pre-trained Language Models Comparison](./03-deep-learning-and-nlp.md#pre-trained-language-models-comparison) | [03](./03-deep-learning-and-nlp.md#pre-trained-language-models-comparison) | +| Masked Language Model | [BERT & Pre-trained Language Models](./03-deep-learning-and-nlp.md#bert--pre-trained-language-models) | [03](./03-deep-learning-and-nlp.md#bert--pre-trained-language-models) | +| Autoregressive | [Pre-trained Language Models Comparison](./03-deep-learning-and-nlp.md#pre-trained-language-models-comparison) | [03](./03-deep-learning-and-nlp.md#pre-trained-language-models-comparison) | +| Self-Attention | [Transformers](./03-deep-learning-and-nlp.md#transformers) — Theory | [03](./03-deep-learning-and-nlp.md#transformers) | +| Multi-Head Attention | [Transformers](./03-deep-learning-and-nlp.md#transformers) — Theory | [03](./03-deep-learning-and-nlp.md#transformers) | +| Positional Encoding | [Transformers](./03-deep-learning-and-nlp.md#transformers) — Theory | [03](./03-deep-learning-and-nlp.md#transformers) | + +### NLP Tasks +| Concept | Section | File | +|---------|---------|------| +| NER | [NER](./03-deep-learning-and-nlp.md#ner-named-entity-recognition) — full section | [03](./03-deep-learning-and-nlp.md#ner-named-entity-recognition) | +| POS Tagging | [POS Tagging](./03-deep-learning-and-nlp.md#pos-tagging) — full section | [03](./03-deep-learning-and-nlp.md#pos-tagging) | +| Sentiment Analysis | [Sentiment Analysis](./03-deep-learning-and-nlp.md#sentiment-analysis) — full section | [03](./03-deep-learning-and-nlp.md#sentiment-analysis) | +| Text Classification | [Text Classification](./03-deep-learning-and-nlp.md#text-classification) — full section | [03](./03-deep-learning-and-nlp.md#text-classification) | +| Machine Translation | [Machine Translation](./03-deep-learning-and-nlp.md#machine-translation) — full section | [03](./03-deep-learning-and-nlp.md#machine-translation) | +| Text Summarization | [Text Summarization](./03-deep-learning-and-nlp.md#text-summarization) — full section | [03](./03-deep-learning-and-nlp.md#text-summarization) | +| Question Answering | [Question Answering](./03-deep-learning-and-nlp.md#question-answering) — full section | [03](./03-deep-learning-and-nlp.md#question-answering) | + +### NLP Evaluation +| Concept | Section | File | +|---------|---------|------| +| Perplexity | [Perplexity](./03-deep-learning-and-nlp.md#perplexity-nlp) — full section | [03](./03-deep-learning-and-nlp.md#perplexity-nlp) | +| BLEU, ROUGE, METEOR | [05-llm-rag-graph.md](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — interview questions | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | +| BLEURT | [05-llm-rag-graph.md](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | + +--- + +## 6. LLMs (Large Language Models) + +| Concept | Section | File | +|---------|---------|------| +| LLM Definition | [LLMs](./05-llm-rag-graph.md#llms-large-language-models) — Theory | [05](./05-llm-rag-graph.md#llms-large-language-models) | +| Foundation Model | [LLMs](./05-llm-rag-graph.md#llms-large-language-models) — Theory | [05](./05-llm-rag-graph.md#llms-large-language-models) | +| Pre-training | [LLMs](./05-llm-rag-graph.md#llms-large-language-models) — Theory | [05](./05-llm-rag-graph.md#llms-large-language-models) | +| Fine-tuning | [Fine-tuning & Adaptation Methods](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | [05](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | +| Instruction Tuning | [Additional LLM Concepts](./05-llm-rag-graph.md#additional-llm-concepts) — Interview Qs | [05](./05-llm-rag-graph.md#additional-llm-concepts) | +| RLHF | [Fine-tuning & Adaptation Methods](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) — Interview Qs (line 238) | [05](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | +| DPO | [Fine-tuning & Adaptation Methods](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) — Theory | [05](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | +| SFT | [Fine-tuning & Adaptation Methods](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) — Example | [05](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | +| ORPO | [Fine-tuning & Adaptation Methods](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) — Theory | [05](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | +| Emergent Abilities | [LLMs](./05-llm-rag-graph.md#llms-large-language-models) — Theory (line 10) | [05](./05-llm-rag-graph.md#llms-large-language-models) | +| In-Context Learning | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory | [05](./05-llm-rag-graph.md#prompt-engineering) | +| Zero-shot / Few-shot | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory | [05](./05-llm-rag-graph.md#prompt-engineering) | +| Chain-of-Thought (CoT) | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory + Interview Qs | [05](./05-llm-rag-graph.md#prompt-engineering) | +| Self-Consistency | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory | [05](./05-llm-rag-graph.md#prompt-engineering) | +| Tree-of-Thoughts (ToT) | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory | [05](./05-llm-rag-graph.md#prompt-engineering) | +| ReAct | [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — Theory | [05](./05-llm-rag-graph.md#agents--tool-use) | +| Context Window | [LLMs](./05-llm-rag-graph.md#llms-large-language-models) — Theory | [05](./05-llm-rag-graph.md#llms-large-language-models) | +| Hallucination | [Evaluation & Hallucination](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — full section | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | +| Scaling Laws (Kaplan, Chinchilla) | [LLMs](./05-llm-rag-graph.md#llms-large-language-models) — Theory | [05](./05-llm-rag-graph.md#llms-large-language-models) | +| Temperature, Top-p, Top-k | [LLMs](./05-llm-rag-graph.md#llms-large-language-models) — Example | [05](./05-llm-rag-graph.md#llms-large-language-models) | +| Logit Bias, Beam Search | [Additional LLM Concepts](./05-llm-rag-graph.md#additional-llm-concepts) | [05](./05-llm-rag-graph.md#additional-llm-concepts) | + +### LLM Inference & Deployment +| Concept | Section | File | +|---------|---------|------| +| KV Cache | Referenced briefly in [LLMs](./05-llm-rag-graph.md#llms-large-language-models) line 16 | [05](./05-llm-rag-graph.md#llms-large-language-models) | +| Speculative Decoding | Referenced briefly in [LLMs](./05-llm-rag-graph.md#llms-large-language-models) line 16 | [05](./05-llm-rag-graph.md#llms-large-language-models) | +| Quantization (GPTQ, AWQ, GGUF) | [LLMs](./05-llm-rag-graph.md#llms-large-language-models) — Theory (line 16) | [05](./05-llm-rag-graph.md#llms-large-language-models) | + +### LLM Architectures +| Model | Section | File | +|-------|---------|------| +| GPT-4, Claude, LLaMA, Mistral, Gemini | [LLMs](./05-llm-rag-graph.md#llms-large-language-models) — Example table | [05](./05-llm-rag-graph.md#llms-large-language-models) | +| Decoding Strategies | [LLMs](./05-llm-rag-graph.md#llms-large-language-models) — Example table | [05](./05-llm-rag-graph.md#llms-large-language-models) | + +--- + +## 7. RAG (Retrieval-Augmented Generation) + +| Concept | Section | File | +|---------|---------|------| +| RAG Definition | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — full section | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| Retriever (Sparse/Dense) | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| Generator | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| Indexing | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| Chunking / Chunk Overlap | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | +| Embedding Model | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| Vector Database | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| ANN (HNSW, IVF, LSH) | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | +| Similarity Search | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| Metadata Filtering | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Example | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| Hybrid Search | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| BM25 | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| Reciprocal Rank Fusion (RRF) | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | +| Query Rewriting / Expansion | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | +| HyDE | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | +| Context Window Management | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| Sliding Window / Parent Document Retriever | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | +| RAPTOR | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | +| Re-ranking | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| Contextual Compression | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | +| Self-RAG, CRAG, Agentic RAG, Graph RAG | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | +| Modular RAG, FLARE | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | +| RAG Pipeline Diagram | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Theory | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| RAG vs Fine-tuning vs Prompt Engineering | [RAG](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) — Example table | [05](./05-llm-rag-graph.md#rag-retrieval-augmented-generation) | +| RAG Failure Modes & Fixes | [Additional RAG Concepts](./05-llm-rag-graph.md#additional-rag-concepts) | [05](./05-llm-rag-graph.md#additional-rag-concepts) | + +--- + +## 8. PROMPT ENGINEERING + +| Concept | Section | File | +|---------|---------|------| +| System / User Prompt | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory | [05](./05-llm-rag-graph.md#prompt-engineering) | +| Few-shot / Zero-shot Prompting | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory | [05](./05-llm-rag-graph.md#prompt-engineering) | +| Chain-of-Thought (CoT) | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory + Interview | [05](./05-llm-rag-graph.md#prompt-engineering) | +| Zero-shot CoT | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory | [05](./05-llm-rag-graph.md#prompt-engineering) | +| Self-Consistency | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory | [05](./05-llm-rag-graph.md#prompt-engineering) | +| Role / Negative / Delimiter Prompting | [Additional Prompt Engineering](./05-llm-rag-graph.md#additional-prompt-engineering) | [05](./05-llm-rag-graph.md#additional-prompt-engineering) | +| Structured Output | [Additional Prompt Engineering](./05-llm-rag-graph.md#additional-prompt-engineering) | [05](./05-llm-rag-graph.md#additional-prompt-engineering) | +| ReAct (Reasoning + Acting) | [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — Theory | [05](./05-llm-rag-graph.md#agents--tool-use) | +| Tree-of-Thoughts (ToT) | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory | [05](./05-llm-rag-graph.md#prompt-engineering) | +| APE (Automatic Prompt Engineer) | [Additional Prompt Engineering](./05-llm-rag-graph.md#additional-prompt-engineering) | [05](./05-llm-rag-graph.md#additional-prompt-engineering) | +| Prompt Chaining | [Additional Prompt Engineering](./05-llm-rag-graph.md#additional-prompt-engineering) | [05](./05-llm-rag-graph.md#additional-prompt-engineering) | +| Best Practices & Templates | [Prompt Engineering](./05-llm-rag-graph.md#prompt-engineering) — Theory | [05](./05-llm-rag-graph.md#prompt-engineering) | + +--- + +## 9. AGENTS & TOOL USE + +| Concept | Section | File | +|---------|---------|------| +| Agent Definition | [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — full section | [05](./05-llm-rag-graph.md#agents--tool-use) | +| Tool / Function Calling | [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — Theory | [05](./05-llm-rag-graph.md#agents--tool-use) | +| Memory (Short/Long-term) | [Additional Agent Concepts](./05-llm-rag-graph.md#additional-agent-concepts) | [05](./05-llm-rag-graph.md#additional-agent-concepts) | +| Planning | [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — Theory | [05](./05-llm-rag-graph.md#agents--tool-use) | +| ReAct Agent | [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — Theory | [05](./05-llm-rag-graph.md#agents--tool-use) | +| Plan-and-Execute | [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — Theory | [05](./05-llm-rag-graph.md#agents--tool-use) | +| Reflection / Self-Evaluation | [Additional Agent Concepts](./05-llm-rag-graph.md#additional-agent-concepts) | [05](./05-llm-rag-graph.md#additional-agent-concepts) | +| Multi-agent / Orchestrator / Router | [Additional Agent Concepts](./05-llm-rag-graph.md#additional-agent-concepts) | [05](./05-llm-rag-graph.md#additional-agent-concepts) | +| Guardrails | [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — Theory | [05](./05-llm-rag-graph.md#agents--tool-use) | +| Human-in-the-Loop | Referenced in [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — Example | [05](./05-llm-rag-graph.md#agents--tool-use) | +| Autonomous / Semi-autonomous Agent | [Additional Agent Concepts](./05-llm-rag-graph.md#additional-agent-concepts) | [05](./05-llm-rag-graph.md#additional-agent-concepts) | +| Agent Architecture Diagram | [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — Theory | [05](./05-llm-rag-graph.md#agents--tool-use) | +| LangGraph, LangChain, AutoGen, CrewAI | [Additional Agent Concepts](./05-llm-rag-graph.md#additional-agent-concepts) | [05](./05-llm-rag-graph.md#additional-agent-concepts) | + +--- + +## 10. GRAPH-BASED ML & KNOWLEDGE GRAPHS + +| Concept | Section | File | +|---------|---------|------| +| Graph / Node / Edge | [Graph-Based ML & Knowledge Graphs](./05-llm-rag-graph.md#graph-based-ml--knowledge-graphs) | [05](./05-llm-rag-graph.md#graph-based-ml--knowledge-graphs) | +| Directed / Undirected / Weighted | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| Adjacency Matrix / Degree | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| Knowledge Graph (KG) | [Graph-Based ML & Knowledge Graphs](./05-llm-rag-graph.md#graph-based-ml--knowledge-graphs) — full section | [05](./05-llm-rag-graph.md#graph-based-ml--knowledge-graphs) | +| Triple (S-P-O) / SPARQL / RDF / OWL | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| Node Embedding / Graph Embedding | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| DeepWalk / Node2Vec | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| GNN / GCN / GAT / GraphSAGE | [Graph-Based ML & Knowledge Graphs](./05-llm-rag-graph.md#graph-based-ml--knowledge-graphs) — Theory | [05](./05-llm-rag-graph.md#graph-based-ml--knowledge-graphs) | +| Message Passing | [Graph-Based ML & Knowledge Graphs](./05-llm-rag-graph.md#graph-based-ml--knowledge-graphs) — Theory | [05](./05-llm-rag-graph.md#graph-based-ml--knowledge-graphs) | +| Graph Transformer / RGCN | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| KGE (TransE, RotatE, ComplEx) | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| Link Prediction / Node / Graph Classification | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| WL Test / Homophily / Heterophily | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| Neo4j / Cypher / PageRank | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| Community Detection | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | +| GraphRAG | [Graph-Based ML & Knowledge Graphs](./05-llm-rag-graph.md#graph-based-ml--knowledge-graphs) — Theory + detailed pipeline | [05](./05-llm-rag-graph.md#graph-based-ml--knowledge-graphs) | +| GoT (Graph-of-Thoughts) | [Additional Graph ML & KG Concepts](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | [05](./05-llm-rag-graph.md#additional-graph-ml--kg-concepts) | + +--- + +## 11. FINE-TUNING & ADAPTATION METHODS + +| Concept | Section | File | +|---------|---------|------| +| Full Fine-tuning | [Fine-tuning & Adaptation Methods](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | [05](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | +| LoRA | [Fine-tuning & Adaptation Methods](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) — Theory | [05](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | +| QLoRA | [Fine-tuning & Adaptation Methods](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) — full Example | [05](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | +| DoRA, Adapter, Prefix Tuning, Prompt Tuning | [Additional Fine-Tuning Concepts](./05-llm-rag-graph.md#additional-fine-tuning-concepts) | [05](./05-llm-rag-graph.md#additional-fine-tuning-concepts) | +| P-Tuning v2, IA³ | [Additional Fine-Tuning Concepts](./05-llm-rag-graph.md#additional-fine-tuning-concepts) | [05](./05-llm-rag-graph.md#additional-fine-tuning-concepts) | +| Catastrophic Forgetting | [Additional Fine-Tuning Concepts](./05-llm-rag-graph.md#additional-fine-tuning-concepts) | [05](./05-llm-rag-graph.md#additional-fine-tuning-concepts) | +| Multi-task Fine-tuning / Domain Adaptation | [Additional Fine-Tuning Concepts](./05-llm-rag-graph.md#additional-fine-tuning-concepts) | [05](./05-llm-rag-graph.md#additional-fine-tuning-concepts) | +| DPO / ORPO | [Fine-tuning & Adaptation Methods](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) — Theory | [05](./05-llm-rag-graph.md#fine-tuning--adaptation-methods) | +| Parameter Efficiency Comparison Table | [Additional Fine-Tuning Concepts](./05-llm-rag-graph.md#additional-fine-tuning-concepts) | [05](./05-llm-rag-graph.md#additional-fine-tuning-concepts) | + +--- + +## 12. EVALUATION & HALLUCINATION IN LLMs + +| Concept | Section | File | +|---------|---------|------| +| BLEU, ROUGE, METEOR, BLEURT | [Evaluation & Hallucination](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — Interview Qs | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | +| Perplexity | [Perplexity](./03-deep-learning-and-nlp.md#perplexity-nlp) — full section | [03](./03-deep-learning-and-nlp.md#perplexity-nlp) | +| Factual Consistency / Faithfulness | [Evaluation & Hallucination](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — Theory | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | +| Hallucination (Intrinsic/Extrinsic) | [Evaluation & Hallucination](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — Theory | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | +| G-Eval | [Evaluation & Hallucination](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — Theory | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | +| LLM-as-Judge | [Evaluation & Hallucination](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — Theory | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | +| Pairwise / Human / A/B Evaluation | [Evaluation & Hallucination](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — Theory | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | +| RAGAS (Context Precision/Recall, Faithfulness, Answer Relevancy) | [Evaluation & Hallucination](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — full Theory | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | +| ARES, TruLens | [Additional Evaluation Concepts](./05-llm-rag-graph.md#additional-evaluation--hallucination-concepts) | [05](./05-llm-rag-graph.md#additional-evaluation--hallucination-concepts) | +| LangSmith / LangFuse | [Agents & Tool Use](./05-llm-rag-graph.md#agents--tool-use) — Interview Qs + [Evaluation](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | [05](./05-llm-rag-graph.md#agents--tool-use) | +| Hallucination Detection (SelfCheckGPT, NLI, Semantic Entropy) | [Evaluation & Hallucination](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — Theory | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | +| Quick Comparison: Which to Use When | [Evaluation & Hallucination](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) — Example | [05](./05-llm-rag-graph.md#evaluation--hallucination-in-llms) | + +--- + +## Status Legend + +| Icon | Meaning | +|------|---------| +| ✅ | Fully covered in existing files | +| 🟡 | Present but scattered / not as dedicated section | +| ❌ | NOT COVERED — needs to be written | + +## Missing Concepts (Priority for Addition) + +| Concept | Recommended File | Priority | +|---------|-----------------|----------| +| CRISP-DM | [01-data-science.md](./01-data-science.md) | High | +| Cross-Validation (dedicated section) | [02-machine-learning.md](./02-machine-learning.md) | High | +| Stratified Cross-Validation | [02-machine-learning.md](./02-machine-learning.md) | High | +| Bias-Variance Tradeoff (dedicated) | [02-machine-learning.md](./02-machine-learning.md) | High | +| Overfitting (dedicated section) | [02-machine-learning.md](./02-machine-learning.md) | High | +| Underfitting (dedicated section) | [02-machine-learning.md](./02-machine-learning.md) | Medium | +| Feature Selection (dedicated) | [02-machine-learning.md](./02-machine-learning.md) | Medium | +| sklearn Pipelines | [02-machine-learning.md](./02-machine-learning.md) | Medium | +| Confusion Matrix (dedicated section) | [02-machine-learning.md](./02-machine-learning.md) | Medium | +| KV Cache (dedicated section) | [05-llm-rag-graph.md](./05-llm-rag-graph.md) | Low | +| Speculative Decoding (dedicated) | [05-llm-rag-graph.md](./05-llm-rag-graph.md) | Low | +| Human-in-the-Loop (dedicated) | [05-llm-rag-graph.md](./05-llm-rag-graph.md) | Low | diff --git a/techy/index.md b/techy/index.md new file mode 100644 index 0000000..1d5e5ae --- /dev/null +++ b/techy/index.md @@ -0,0 +1,30 @@ +# EY L1 Data Science Interview — Complete Study Guide + +## File Index + +| # | Section | File | +|---|---------|------| +| 1 | Data Science (General) | [01-data-science.md](./01-data-science.md) | +| 2 | Machine Learning | [02-machine-learning.md](./02-machine-learning.md) | +| 3 | Deep Learning | [03-deep-learning.md](./03-deep-learning.md) | +| 4 | Natural Language Processing | [04-nlp.md](./04-nlp.md) | +| 5 | LLMs, RAG, Graph, Prompt Engineering, Agents, Fine-tuning, Evaluation | [05-llm-rag-graph.md](./05-llm-rag-graph.md) | + +## Format Per Concept + +``` +## Concept Name +**Definition:** 1-2 sentence crisp definition + +### Theory & Explanation +3-5 paragraphs covering how it works, why it matters, mathematical intuition, key properties + +### Example +Concrete example with numbers, code snippet, or scenario + +### Interview Questions +2-3 likely questions with answers + +### Related Concepts +Cross-references to other concepts in the guide +```