01
PROJECT OVERVIEW

Project at a glance

Project overview

KrishiSignal collects daily mandi price records and presents them through a data pipeline and Streamlit dashboard for agricultural price intelligence.

Implementation

I built the pipeline with Python, Kafka, PySpark and HDFS, with a Streamlit dashboard. It requests the latest available daily records from the data.gov.in API; the source data is updated daily, rather than as a continuous live feed.

02
DIRECTION

Project objectives

  • Request the latest available daily mandi records from the Data.gov.in resource.

  • Process records through the documented Python, Kafka and PySpark flow.

  • Present project data in a Streamlit dashboard.

03
CAPABILITIES

Project features

01

Mandi data requests

Uses the current daily prices resource from data.gov.in.

02

Data pipeline

Python producer, Kafka, PySpark/Spark Structured Streaming and HDFS.

03

Dashboard

Streamlit interface for the project’s market data and analysis.

04
TECHNICAL DESIGN

Project data flow

Pipeline technologies and operation are described by the developer. Data source records are published daily.

  1. 01

    Data.gov.in API

    Latest available daily mandi records

  2. 02

    Python

    Producer

  3. 03

    Kafka

    Message stream

  4. 04

    PySpark

    Stream processing

  5. 05

    HDFS

    Data storage

  6. 06

    Streamlit

    Dashboard

05
PROCESS

Development workflow

  1. 01

    Request the latest available daily records from the API.

  2. 02

    Pass records through the Python producer and Kafka.

  3. 03

    Process records with PySpark and store project output in HDFS.

  4. 04

    Explore the resulting data through Streamlit.

06
VISUALS

Project gallery

Captions identify loading states and dated captures where relevant.

07
REFLECTION

Challenges & learning focus

Challenges

  • Connect multiple data-processing tools into a working project flow.
  • Communicate the source’s daily update cadence clearly.

Learnings

  • Data freshness should be described according to the source update frequency.
  • A pipeline becomes easier to understand when each system’s role is explicit.