Work Hard EverywhereJob Description
Headquarters: Remote
URL: https://www.toptal.com/
About the Role
We're looking for a Senior Data/ML Engineer to refactor, operationalize, and improve an existing time series forecasting platform that's already live and driving real business value. This is not a greenfield build - the focus is modernizing a Databricks-based forecasting system that's been maintained primarily by a single developer for years. You'll reduce technical debt, strengthen testing and observability, and raise the engineering bar on a system the business already depends on. If you'd rather bring discipline and maturity to an existing production system than start from a blank slate, this is built for that.
What You'll Do
-
Review the current forecasting platform architecture and identify areas for improvement
-
Refactor existing Databricks, Python, and PySpark implementations
-
Move business logic out of Databricks notebooks and into reusable Python modules or packages
-
Improve separation of concerns between orchestration and core business logic
-
Establish stronger engineering standards and help define what "good" looks like for the platform
-
Implement or improve automated testing practices and validation mechanisms for forecasting workflows
-
Build or improve monitoring and observability, increasing visibility into how predictions are generated
-
Help monitor model behavior and operational health over time
-
Improve reliability of scheduled training workflows, reducing manual intervention on failure
-
Improve failure handling, retries, and overall workflow resilience
-
Maintain and extend existing forecasting capabilities as needed
What You Bring
-
Strong professional experience with Databricks, including workspaces, notebooks, scheduled workflows, and CI/CD processes
-
Strong Python engineering experience, including designing reusable modules or packages
-
Strong PySpark experience with production data pipelines or distributed data processing
-
Experience refactoring...
URL: https://www.toptal.com/
About the Role
We're looking for a Senior Data/ML Engineer to refactor, operationalize, and improve an existing time series forecasting platform that's already live and driving real business value. This is not a greenfield build - the focus is modernizing a Databricks-based forecasting system that's been maintained primarily by a single developer for years. You'll reduce technical debt, strengthen testing and observability, and raise the engineering bar on a system the business already depends on. If you'd rather bring discipline and maturity to an existing production system than start from a blank slate, this is built for that.
What You'll Do
-
Review the current forecasting platform architecture and identify areas for improvement
-
Refactor existing Databricks, Python, and PySpark implementations
-
Move business logic out of Databricks notebooks and into reusable Python modules or packages
-
Improve separation of concerns between orchestration and core business logic
-
Establish stronger engineering standards and help define what "good" looks like for the platform
-
Implement or improve automated testing practices and validation mechanisms for forecasting workflows
-
Build or improve monitoring and observability, increasing visibility into how predictions are generated
-
Help monitor model behavior and operational health over time
-
Improve reliability of scheduled training workflows, reducing manual intervention on failure
-
Improve failure handling, retries, and overall workflow resilience
-
Maintain and extend existing forecasting capabilities as needed
What You Bring
-
Strong professional experience with Databricks, including workspaces, notebooks, scheduled workflows, and CI/CD processes
-
Strong Python engineering experience, including designing reusable modules or packages
-
Strong PySpark experience with production data pipelines or distributed data processing
-
Experience refactoring...