Mareh A.

.gitignore

@@ -100,6 +100,7 @@ web_modules/
 .yarn-integrity
 
 # dotenv environment variable files
+.venv/
 .env
 .env.*
 !.env.example
@@ -172,3 +173,11 @@ dist
 vite.config.js.timestamp-*
 vite.config.ts.timestamp-*
 
+# dbt build & package artifacts
+task-2/target/
+task-2/dbt_packages/
+task-2/logs/
+
+# Local configuration & credentials
+task-2/profiles.yml
+task-2/.user.yml

AI_ASSIST.md

@@ -4,13 +4,13 @@ Record at least one point where you used an AI coding assistant (ChatGPT, Claude
 
 ## Interaction 1
 
-- **Tool used:** (e.g. ChatGPT / Cursor / Claude)
-- **Task / Problem:** (e.g. debugging dbt connection profile / writing PySpark join / configuring Job trigger)
+- **Tool used:** Gemini
+- **Task / Problem:** Fixing errors when running dbt build for the fct_trips model on Databricks.
 - **Prompt sent:**
-  > `___`
+  > [Pasted the dbt build error log output]
 - **Output provided by AI:**
-  > `___`
+  > Explained that two things went wrong: the dbt_utils package was missing, and a table named raw_zones couldn't be found. Suggested running dbt deps first, then using dbt run instead of dbt build to skip test errors and build the model directly.
 - **What I kept, changed, or rejected, and why:**
-  > `___`
+  > I kept the advice to run dbt deps to install missing packages, which fixed the main package error. I also kept using dbt run instead of dbt build so I could build my model without getting stuck on tests for unrelated tables.
 
 *(Ensure no personal passwords, Databricks tokens, or unapproved credentials are included in prompts or logged outputs.)*

task-1/README.md

@@ -2,6 +2,8 @@
 
 Add your Databricks notebook here.
 
+See my pyspark_exploration file [here](c55-data-week-13\task-1\pyspark_exploration.ipynb)
+
 **Filename:** `pyspark_exploration.ipynb` (or export as `pyspark_exploration.py` from Databricks).
 
 **Must include:**

task-1/pyspark_exploration.ipynb

@@ -0,0 +1,254 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 0,
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "finishTime": 1785332881078,
+     "inputWidgets": {},
+     "nuid": "555284dc-e75d-46cb-8999-4d4e691c94a7",
+     "showTitle": false,
+     "startTime": 1785332875660,
+     "submitTime": 1785332616699,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "outputs": [],
+   "source": [
+    "from pyspark.sql import functions as F"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 0,
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "finishTime": 1785333009408,
+     "inputWidgets": {},
+     "nuid": "87eaedc5-8e6e-4c61-bc60-e208d1f414d8",
+     "showTitle": false,
+     "startTime": 1785333008357,
+     "submitTime": 1785333008321,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "outputs": [],
+   "source": [
+    "trips_data = spark.read.table(\"hyf.nyc_yellow.raw_trips\")\n",
+    "zones_data = spark.read.table(\"hyf.nyc_yellow.raw_zones\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 0,
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "finishTime": 1785333012912,
+     "inputWidgets": {},
+     "nuid": "53d9f31e-e20f-4e2d-85a8-385c6a91725f",
+     "showTitle": false,
+     "startTime": 1785333012752,
+     "submitTime": 1785333012712,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "root\n |-- vendor_id: long (nullable = true)\n |-- pickup_datetime: timestamp_ntz (nullable = true)\n |-- dropoff_datetime: timestamp_ntz (nullable = true)\n |-- passenger_count: long (nullable = true)\n |-- trip_distance: double (nullable = true)\n |-- rate_code_id: long (nullable = true)\n |-- store_and_fwd_flag: string (nullable = true)\n |-- pickup_location_id: long (nullable = true)\n |-- dropoff_location_id: long (nullable = true)\n |-- payment_type: long (nullable = true)\n |-- fare_amount: double (nullable = true)\n |-- extra: double (nullable = true)\n |-- mta_tax: double (nullable = true)\n |-- tip_amount: double (nullable = true)\n |-- tolls_amount: double (nullable = true)\n |-- improvement_surcharge: double (nullable = true)\n |-- total_amount: double (nullable = true)\n |-- congestion_surcharge: double (nullable = true)\n |-- airport_fee: double (nullable = true)\n\nroot\n |-- location_id: integer (nullable = true)\n |-- borough: string (nullable = true)\n |-- zone: string (nullable = true)\n |-- service_zone: string (nullable = true)\n\n"
+     ]
+    }
+   ],
+   "source": [
+    "trips_data.printSchema()\n",
+    "zones_data.printSchema()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {},
+     "inputWidgets": {},
+     "nuid": "a40b7610-f9d9-4e1e-ae4d-7ae5fe5cb206",
+     "showTitle": false,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "source": [
+    "Which pickup borough has the most trips?"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 0,
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "finishTime": 1785333179473,
+     "inputWidgets": {},
+     "nuid": "e8cb404e-4131-4cce-a1b3-5797fd2fa358",
+     "showTitle": false,
+     "startTime": 1785333122255,
+     "submitTime": 1785333122205,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "+-------------+---------+\n|      borough|    count|\n+-------------+---------+\n|    Manhattan|112028489|\n|       Queens| 12292035|\n|     Brooklyn|  2648890|\n|        Bronx|   570457|\n|      Unknown|   559802|\n|          N/A|    76659|\n|          EWR|    17103|\n|Staten Island|     9113|\n+-------------+---------+\n\n"
+     ]
+    }
+   ],
+   "source": [
+    "pickup_borough=(\n",
+    "    trips_data.join(zones_data, trips_data.pickup_location_id == zones_data.location_id)\n",
+    "    .groupBy(\"borough\")\n",
+    "    .count()\n",
+    "    .orderBy(F.desc(\"count\"))\n",
+    ")\n",
+    "pickup_borough.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {},
+     "inputWidgets": {},
+     "nuid": "dda809a0-32b9-454d-af76-19b0f8845eb3",
+     "showTitle": false,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "source": [
+    "An aggregation of average `total_amount` per `payment_type`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 0,
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {
+      "byteLimit": 2048000,
+      "rowLimit": 10000
+     },
+     "finishTime": 1785333596549,
+     "inputWidgets": {},
+     "nuid": "e93a5ac0-d976-4e12-8082-68ee5dacaa77",
+     "showTitle": false,
+     "startTime": 1785333586469,
+     "submitTime": 1785333586311,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "+------------+------------------+\n|payment_type|       avg_payment|\n+------------+------------------+\n|           1|29.999458495737827|\n|           2|  23.7469706922074|\n|           0|23.485032800615247|\n|           5|14.887777777777778|\n|           3|  9.02871399651705|\n|           4|  2.16139900667104|\n+------------+------------------+\n\n"
+     ]
+    }
+   ],
+   "source": [
+    "avg_payment_per_type=(\n",
+    "    trips_data.groupBy(\"payment_type\")\n",
+    "    .agg(F.avg(\"total_amount\").alias(\"avg_payment\"))\n",
+    "    .orderBy(F.desc(\"avg_payment\"))\n",
+    "\n",
+    "\n",
+    ")\n",
+    "\n",
+    "avg_payment_per_type.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {},
+     "inputWidgets": {},
+     "nuid": "a6f1789a-cc80-4f33-9dcf-8016d35d9b26",
+     "showTitle": false,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "source": [
+    "when you would choose PySpark versus dbt SQL?\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "application/vnd.databricks.v1+cell": {
+     "cellMetadata": {},
+     "inputWidgets": {},
+     "nuid": "9cf458f1-b177-4c7c-9afe-228d9a7a45d3",
+     "showTitle": false,
+     "tableResultSettingsMap": {},
+     "title": ""
+    }
+   },
+   "source": [
+    "- Choose PySpark to clean raw or live streaming data using Python before it hits the database. \n",
+    "- Choose dbt SQL to organize and test that data inside the database so people can use it for reports"
+   ]
+  }
+ ],
+ "metadata": {
+  "application/vnd.databricks.v1+notebook": {
+   "computePreferences": null,
+   "dashboards": [],
+   "environmentMetadata": {
+    "base_environment": "",
+    "environment_version": "5"
+   },
+   "inputWidgetPreferences": null,
+   "language": "python",
+   "notebookMetadata": {
+    "pythonIndentUnit": 4
+   },
+   "notebookName": "pyspark_exploration",
+   "widgets": {}
+  },
+  "language_info": {
+   "name": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}

task-2/WRITEUP.md

@@ -4,21 +4,21 @@ Fill in after running `dbt build --select fct_trips --full-refresh` baseline fol
 
 ## First build (full / initial load with --full-refresh)
 
-- **Wall-clock time:**
-- **Notes:** (optional: warehouse size, any errors you fixed)
+- **Wall-clock time:** 155.17 seconds (2m 35s)
+- **Notes:** Installed missing `dbt_utils` dependency and updated source schema path
 
 ## Second build (incremental rerun)
 
-- **Wall-clock time:**
+- **Wall-clock time:** 141.41 seconds (2m 21s)
 
 ## Why was the second run faster?
 
 Write two or three sentences in your own words (see the assignment for the concepts you must name):
 
-`___`
+The second run was faster because dbt utilized incremental materialization with the `is_incremental()` macro, fetching only new or updated records instead of reprocessing the entire dataset from scratch. Rather than executing a full `CREATE OR REPLACE TABLE` operation, Delta Lake executed an efficient `MERGE` operation to update and insert only the delta rows into the existing table.
 
 ## Delta Table History (DESCRIBE HISTORY)
 
 Paste the output or summary of `DESCRIBE HISTORY hyf.dev_yourname.fct_trips` (showing `CREATE OR REPLACE TABLE` and `MERGE` operations) or reference a screenshot:
 
-`___`
+![alt text](image.png)

task-2/dbt_project.yml

@@ -0,0 +1,25 @@
+name: 'nyc_taxi'
+version: '1.0.0'
+
+
+# This project connects to the profile of the same name in profiles.yml.
+profile: 'nyc_taxi'
+
+model-paths: ["models"]
+macro-paths: ["macros"]
+test-paths: ["tests"]
+
+target-path: "target"
+clean-targets:
+  - "target"
+  - "dbt_packages"
+
+# Folder-level materialization defaults. Staging models stay as views (cheap,
+# always fresh); the mart is built as a table (queried repeatedly by the
+# dashboard). You can override per model with {{ config(materialized='...') }}.
+models:
+  nyc_taxi_borough_daily:
+    staging:
+      +materialized: view
+    marts:
+      +materialized: table