✨ Add dag to begin to mint global ids#46
Conversation
| from airflow.sdk.bases.hook import BaseHook | ||
| except ( | ||
| ImportError | ||
| ): # Since Airflow 3.1, the BaseHook is in the airflow.sdk.bases.hook module |
There was a problem hiding this comment.
while we are currently on airflow 3.0.6, mwaa added airflow v3.2.1 support, so if we upgrade, this will allow for that upgrade. docs on mwaa airflow supported versions
| enum=[ | ||
| "T3JnYW5pemF0aW9uOmNtZjJ3bzlrdDAwMG9rMTAxcHd4cHFmMWQ=", | ||
| "T3JnYW5pemF0aW9uOmNsZHN4MzRrbjAwMTRnMGVzY3JndzUzYWQ=", | ||
| "org_id_2", |
There was a problem hiding this comment.
I need the org ID for include dewrangle org here and in values_display
There was a problem hiding this comment.
T3JnYW5pemF0aW9uOmNsZWhibTF4ZjAwZTdpY2VzZjI0d2tlNHk=
| }, | ||
| ), | ||
| }, | ||
| tags=["non_program", "id_minting", "dewrangle"], |
There was a problem hiding this comment.
are there any other tags folks would want for this?
| postgres_conn = BaseHook.get_connection("postgres_prd_svc") | ||
| dewrangle_conn = BaseHook.get_connection("dewrangle_api") |
There was a problem hiding this comment.
These are both airflow connections that are maintained within airflow, holding connection secrets
| "PROD_DCC_WAREHOUSE_DEWRANGLE_IDS_SCHEMA": "{{ params.globalid_schema_name }}", | ||
| "PROD_DCC_WAREHOUSE_DEWRANGLE_IDS_TABLE": "{{ params.globalid_table_name }}", | ||
| "DCC_WAREHOUSE_HOST": postgres_conn.host, | ||
| "DCC_WAREHOUSE_PORT": str(postgres_conn.port or 5432), |
There was a problem hiding this comment.
airflow only allows passing environment variables as strings, which is why this is coerced to string and DEWRANGLE_CLIENT_EXECUTION_TIMEOUT is a string.
| [connect-to-airflow.md](/Users/friedmanc1/Documents/include-dbt-sandbox/pipeline_docs/guides/connect-to-airflow.md). | ||
| 2. The DAG `global_id_minting` is available in Airflow. | ||
| 3. Your source table already exists in the warehouse. | ||
| 4. You know which warehouse schema and table should receive the minted global |
There was a problem hiding this comment.
Does the target table need to already exist? Or does this create them as well
There was a problem hiding this comment.
good catch, updating to reflect that it doesn't nee to exist
|
I left the comments, but otherwise, it looks good to me. |
4 participants
Add DAG to mint global ids
Adds a dag and associated guide on its use, for minting global ids using airflow
Closes D3B-2873
Type of change
Please delete options that are not relevant.
to not work as expected)
How Has This Been Tested?
running in mwaa
Checklist
Please check all of the items below before merging this pull request. If an
item in the list below does not need to be completed, please indicate the
reason why.
the start of each script
works
sqlfluff fixon tables that are created or modified in this PR.