QSIM — Orchestration & Sentiment Pipeline

Overview

QSIM (Quant Signal Manager) is an internal execution tool for working with trading signals. It brings together manually trained quant models, social media sentiment signals from Reddit, and a fast data layer on ClickHouse, so that researchers and downstream systems can query, combine and act on enriched signals at scale.

My job was the end-to-end data pipelines: Reddit ingestion, LLM-based sentiment analysis, asset tagging, storage in ClickHouse, and the core libraries that make all of that easy to consume.

Data pipelines and orchestration

• Designed Prefect 2 flows for the whole path from Reddit to sentiment to signals
• Implemented ETL steps for:
  • fetching and normalizing Reddit posts and comments,
  • running LLM-based sentiment classification,
  • recognizing assets through keyword and tag rules,
  • writing clean, analytics-ready records into ClickHouse.
• Exposed the flows as reusable building blocks for different assets and universes, so a new signal set could be onboarded with very little code

Cost-efficient Reddit parsing

Earlier systems leaned on large proxy pools to collect Reddit data. That was expensive and it broke often.

I built a new Reddit parsing system that took proxy costs to zero. Instead of rotating proxies, it uses several legitimate bot accounts set up within Reddit's policies and pools their rate limits. A small custom load balancer tracks each account's current usage, picks which one handles the next request, and keeps throughput high without any single account hitting its limit.

The result was no proxy spend at all, more predictable behaviour with fewer parsing failures, and less infrastructure to operate.

Multi-repo core and data libraries

QSIM was split across several repositories:

• qsim-core for shared domain logic and interfaces
• qsim-executor for execution tooling and orchestration
• qsim-data for data access, models and ETL utilities

I owned the core and data modules and the job of making them integrate cleanly across repos. qsim-core is imported by both of the others, and it defines the reusable abstractions: signal definitions and metadata, dataset schemas and transformations, and the shared utilities that flows and services rely on.

High-performance data access (ClickHouse and SQLAlchemy)

Downstream tools and notebooks had to work with large volumes of signal data without much ceremony.

• Implemented data access interfaces with sqlalchemy for relational sources and clickhouse-connect for ClickHouse
• Exposed simple methods that return raw or aggregated signals, time-window slices, and joined model and sentiment data as Pandas or PyArrow DataFrames, ready for research and backtesting
• Tuned schemas and queries so large result sets come back efficiently, without boilerplate repeated in every project

Impact

Proxy costs went to zero once Reddit collection moved to multiple bot accounts behind the custom load balancer. The sentiment and asset tagging pipeline feeds ClickHouse with clean, queryable data that combines with quant model outputs. And the core and data libraries, plus the simple data access APIs, made new flows, signals and research tooling much quicker to build on top of QSIM.