Job Title: Senior Analytics Engineer (Databricks & PySpark)
Location: Chicago, IL (Hybrid – 1 day/week on Tuesdays at Willis Tower)
Pay Rate: Up to $65/hr (Depending on experience)
Position Overview
We are seeking an expert Senior Analytics Engineer for a short-term contract to accelerate the modernization and optimization of our data platform footprint. Our team sits at the critical intersection of customer digital journeys, platform reliability, and operational performance telemetry.
Your primary mission will be to dive into our existing Redshift and sandbox data environments, untangle complex data structures, and use Databricks and PySpark to refactor, optimize, and formally architect these datasets into highly efficient, scalable, and production-grade Analytics Data Products.
The ideal candidate requires minimal onboarding, possesses deep technical mastery of log-parsing and clickstream data, and can implement strict data governance standards from day one. You will also serve as a key technical partner, creating scalable blueprints that can be seamlessly handed off to core analytics, broader engineering, and offshore teams for long-term maintenance.
Core Responsibilities
-
Data Product Architecture: Transform ad-hoc sandbox queries, clickstream event data, and massive transactional log sources into structured, reliable, downstream analytics data assets.
-
Pipeline Refactoring & Optimization: Convert legacy SQL/Redshift logic into highly optimized, modular PySpark pipelines within Databricks, drastically reducing compute times and improving query efficiency.
-
Log Parsing & Behavioral Analytics: Standardize the ingestion and structural mapping of diverse semi-structured log data (e.g., shopping cart logs, API error events, system performance logs) to build precise conversion funnels, attribution logic, and system reliability metrics.
-
Governance & Standardization: Enforce rigorous technical standards across all data products, ensuring strict schema enforcement, detailed metadata documentation, and standardized lowercase/underscore naming conventions across all outputs.
-
Production Blueprinting & Handoff: Partner with the core analytics team to transition experimental sandbox work into robust, production-ready assets designed for long-term support and smooth handoff to engineering and offshore maintenance teams.
Technical Requirements & Qualifications
-
Expert PySpark & Databricks: 5+ years of hands-on experience designing, optimizing, and maintaining large-scale data processing workloads in Databricks using PySpark (DataFrames, Structured Streaming, Delta Lake).
-
Advanced SQL & Redshift Optimization: Deep expertise writing and optimizing complex, high-volume queries in AWS Redshift environments.
-
Legacy Refactoring Mastery: Proven track record of auditing unfamiliar database schemas, reverse-engineering complex legacy SQL/PySpark logic, and refactoring it into clean, modular, production-grade code.
-
Analytics Domain Context: Strong understanding of optimizing data structures specifically for analytics use cases (e.g., behavioral analytics, conversion funnels, attribution logic, and performance monitoring).
-
Immediate Autonomy & Communication: Self-starter mindset with the ability to operate independently with minimal business context, while maintaining strong collaborative partnerships with core analytics and offshore engineering teams.
Preferred Qualifications
-
Prior experience in fast-paced, high-volume, enterprise-scale environments (e.g., Travel, E-Commerce, or Logistics industries).
-
Familiarity with transactional log sources and complex clickstream data structures.
-
Experience creating documentation and standardized templates for offshore team handoffs.