This page describes the AI Engine service and the Airflow-based data pipelines in the repository.
The AI Engine is a Flask application defined under services/ai_engine/:
app.py — main Flask app that exposes HTTP endpoints for AI-powered features (e.g., chat or analysis endpoints) and integrates with external APIs such as OpenAI (per the root README.md).constants.py — shared configuration constants.templates/ — HTML templates used by the app.Dockerfile — container image definition used by the docker-compose.dev.yml dockerllm service.In the development docker-compose topology:
dockerllm with container ai_engine.5001 and 5002 are exposed to the host..env; mounts local speech data into /data/speech_data.The AI Engine can be used standalone for experimentation or wired into other applications as an upstream AI microservice.
The Airflow system is defined under services/airflow/ and uses DAG code colocated under the top-level airflow/dags/ directory.
Key DAGs (from the root README.md and airflow/dags/*):
fetch_stock_data.py — downloads historical stock prices using yfinance.fetch_mutual_fund_etf_data.py — retrieves Indian mutual fund and ETF data from yfinance.fetch_nifty50_stock_data.py and nifty50_ai_stock_data_download.py — fetch data for the Nifty 50 index and store it in S3.homepage_indices_graph_data.py — generates data used by application home page graphs.post_to_social.py — posts updates to social channels.user_portfolio_processor.py — processes user portfolio data.airflow-db-cleanup.py — periodic cleanup for the Airflow metadata database.Helper modules such as s3_utils.py and stock_constants.py under airflow/dags/ and utils/common provide shared logic for S3 access and constant definitions.
In production-like configurations (commented in docker-compose.prod.yml):
webserver, scheduler, worker, and flower services can be enabled by uncommenting the relevant sections and building from services/airflow/Dockerfile.services/airflow/dags and log directories from services/airflow/airflow_logs.Kubernetes manifests under infrastructure/k8s-manifests can be used to run Airflow in a cluster for higher reliability.
When modifying DAGs or the AI Engine, always consider: