https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/deploying-dj/Recent content in Deploying DJ onHugo -- gohugo.iohttps://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/overview/Mon, 01 Jan 0001 00:00:00 +0000https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/overview/The Components of a DJ Deployment # A full DJ deployment consists of the following services: Core API UI Postgres DB (for storing metadata) Query Service Reflection Service Materialization Service (optional) flowchart LR classDef coreAPI fill:#ffc9c9,stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef UI fill:#99e9f2,stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef db fill:#b2f2bb,stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef reflection fill:#d0bfff,stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef query fill:#ffec99,stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef clients fill:#e3fafc,stroke:#000,stroke-width:2px,color:#000,font-size:22px subgraph DataLayer["Data Layer"] direction TB DataWarehouse["Data Warehouse"] Scheduler["Scheduler"] end subgraph "DataJunction" direction TB CoreAPI("<br>  Core API  <br><br>"):::coreAPI MetadataDB["Metadata DB<br>(Postgres)"]:::db QueryService["Query<br>Service"]:::query ReflectionService["Reflection<br>Service"]:::reflection subgraph Clients["Client Libraries"] PythonClient["Python Client"]:::clients JSClient["JS Client"]:::clients end UI("<br>  DJ UI  <br><br>"):::UI end subgraph "Presentation Layer" direction TB JupyterNotebooks["Jupyter Notebooks"] Streamlit["Streamlit"] ReactApps["React Apps"] Tableau["Tableau"] end CoreAPI <--> MetadataDB CoreAPI <--> UI CoreAPI --> PythonClient["Python Client"] CoreAPI --> JSClient["JS Client"] DataWarehouse --> QueryService Scheduler --> QueryService ReflectionService --> CoreAPI QueryService <--> CoreAPI DataWarehouse --> ReflectionService PythonClient --> JupyterNotebooks["Jupyter Notebooks"] PythonClient --> Streamlit["Streamlit"] JSClient --> ReactApps["React Apps"]https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/running-a-dj-server/Mon, 01 Jan 0001 00:00:00 +0000https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/running-a-dj-server/Running the core DJ Server # This guide walks through configuring and running a DJ (DataJunction) server. Configuration # DJ uses Pydantic to manage configuration, with support for environment variables (including nested variables using __ as a delimiter). The Settings class controls server behavior, and each setting can be overridden via environment variables or configuration files. Below is a list of all useful server settings. Core Server Settings # Setting Type Default Description query_service str | None None Optional external query service URL.https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/query-service/Mon, 01 Jan 0001 00:00:00 +0000https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/query-service/flowchart LR classDef coreAPI stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef UI stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef db stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef reflection stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef query fill:#ffec99,stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef clients stroke:#000,stroke-width:2px,color:#000,font-size:22px subgraph DataLayer["Data Layer"] direction TB DataWarehouse["Data Warehouse"] Scheduler["Scheduler"] end subgraph "DataJunction" direction TB CoreAPI("<br>  Core API  <br><br>"):::coreAPI MetadataDB["Metadata DB<br>(Postgres)"]:::db QueryService["<br>  Query  <br>  Service  <br><br>"]:::query ReflectionService["Reflection<br>Service"]:::reflection subgraph Clients["Client Libraries"] PythonClient["Python Client"]:::clients JSClient["JS Client"]:::clients end UI("<br>  DJ UI  <br><br>"):::UI end subgraph "Presentation Layer" direction TB JupyterNotebooks["Jupyter Notebooks"] Streamlit["Streamlit"] ReactApps["React Apps"] Tableau["Tableau"] end CoreAPI <--> MetadataDB CoreAPI <--> UI CoreAPI --> PythonClient["Python Client"] CoreAPI --> JSClient["JS Client"] DataWarehouse --> QueryService Scheduler --> QueryService ReflectionService --> CoreAPI QueryService <--> CoreAPI DataWarehouse --> ReflectionService PythonClient --> JupyterNotebooks["Jupyter Notebooks"] PythonClient --> Streamlit["Streamlit"] JSClient --> ReactApps["React Apps"] The query service is responsible for the execution of data retrieval queries against the underlying data sources.https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/reflection-service/Mon, 01 Jan 0001 00:00:00 +0000https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/reflection-service/flowchart LR classDef coreAPI stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef UI stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef db stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef reflection fill:#ffec99,stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef query stroke:#000,stroke-width:2px,color:#000,font-size:22px classDef clients stroke:#000,stroke-width:2px,color:#000,font-size:22px subgraph DataLayer["Data Layer"] direction TB DataWarehouse["Data Warehouse"] Scheduler["Scheduler"] end subgraph "DataJunction" direction TB CoreAPI("<br>  Core API  <br><br>"):::coreAPI MetadataDB["Metadata DB<br>(Postgres)"]:::db QueryService["Query<br>Service"]:::query ReflectionService["<br>  Reflection  <br>Service <br><br>"]:::reflection subgraph Clients["Client Libraries"] PythonClient["Python Client"]:::clients JSClient["JS Client"]:::clients end UI("<br>  DJ UI  <br><br>"):::UI end subgraph "Presentation Layer" direction TB JupyterNotebooks["Jupyter Notebooks"] Streamlit["Streamlit"] ReactApps["React Apps"] Tableau["Tableau"] end CoreAPI <--> MetadataDB CoreAPI <--> UI CoreAPI --> PythonClient["Python Client"] CoreAPI --> JSClient["JS Client"] DataWarehouse --> QueryService Scheduler --> QueryService ReflectionService --> CoreAPI QueryService <--> CoreAPI DataWarehouse --> ReflectionService PythonClient --> JupyterNotebooks["Jupyter Notebooks"] PythonClient --> Streamlit["Streamlit"] JSClient --> ReactApps["React Apps"] The reflection service polls the DJ core service for all nodes with associated tables, whether source tables or materialized tables.https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/caching/Mon, 01 Jan 0001 00:00:00 +0000https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/caching/In DataJunction, caching is a crucial component that helps optimize performance by storing and reusing results of expensive operations, such as computing the dimension DAG (Directed Acyclic Graph). This section discusses how caching is used within DataJunction and how you can implement a custom caching solution using FastAPI’s dependency injection. How Caching is Used # DataJunction employs caching in multiple areas to enhance performance and reduce the load on the database.https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/sql-plugins/Mon, 01 Jan 0001 00:00:00 +0000https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/sql-plugins/DataJunction has a flexible SQL transpilation plugin system that enables you to add support for transpiling SQL queries across any SQL dialects. By default, support is enabled for Spark SQL, Trino, and Druid. You can also add custom transpilation plugins by following the instructions below. 1. Create the Plugin Class Subclass SQLTranspilationPlugin, and register it with the dialect(s) it supports using the @dialect_plugin decorator: @dialect_plugin("customdb") class CustomDBTranspilationPlugin(SQLTranspilationPlugin): package_name: str = "customdb-sql" def transpile_sql( self, query: str, *, input_dialect: Optional[Dialect] = None, output_dialect: Optional[Dialect] = None, ) -> str: import customdb_sql # Ensure the library is installed # Ex: use the custom library’s transpilation interface return customdb_sql.https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/notifications/Mon, 01 Jan 0001 00:00:00 +0000https://6a024bf2a7e6a60008bd1284--thriving-cassata-78ae72.netlify.app/docs/0.1.0/deploying-dj/notifications/In DataJunction, notifications are crucial for keeping users informed about various activities, such as changes in the state of nodes. This document outlines how notifications are managed within DataJunction and how you can implement a custom notification solution using FastAPI’s dependency injection. How Notifications are Used # DataJunction uses notifications to alert users about significant events, such as updates to nodes or changes in their state. By handling notifications within the OSS project, DataJunction provides an opinionated take on where and when notifications should be sent.