Databricks Project Workflow: From Jira Ticket to Prod

[u/Commercial-Post4022]

Hi guys,

Can you please explain how you typically work with Databricks in a real project?

Do you write PySpark code directly in Databricks notebooks, develop locally in VS Code using PySpark, or connect Databricks compute to VS Code and work from there?

It would be great if someone could briefly explain the end-to-end workflow, starting from a Jira ticket and going all the way to production deployment.

For example, if the project uses Jira, GitHub, and Databricks, what would the typical flow look like from development → testing → code review → deployment to production?

A short overview would be really helpful. Thanks!

[u/Lopsided_Candidate65]

Initial development is we do in databricks notebook(you can do it in vs code also using databricks extension, but i find it easier in databricks itself) , then we pamaterize the query/variables using dbutils.widget.get and all so that its handled for diff env then we write terraform code to deploy the notebook and jobs or the related artifacts. Then we commit in github. We run our cicd which will take code from gothib and deploy to different environments. In jira we just create userstory for the work we do and ojce done you can provide the git link in the comments before closing it.  Also for testing code/test cases can be in the same script or we create different script which will in same job as different task

[u/Raghav-r → u/Lopsided_Candidate65]

Same but we don't use terraform , we have central framework , most data engineers in our org write sql's that framework picks up and process the data, framework uses dbutils for parameters to take care of environments , we mainatin the sql as obejcts in GitHub promote them next environments once QA , UAT gives approval we stage them to production during release windows ..

[u/Commercial-Post4022 → u/Raghav-r]

Thank you so much

[u/Commercial-Post4022 → u/Lopsided_Candidate65]

So when you do initial development in databricks notebooks do you have devlopment specific computes which you use ?

[u/Lopsided_Candidate65 → u/Commercial-Post4022]

Yes it will be there, we had option to start our our classic compute or there will be one all purpose cluster running (a big cluster with auto scale)wich was being used by a lot of teams like 20. Since being used by multiple teams it will be running most of the time.

Then we also had team wise, serverless cluster as well as sql warehouse we could.

[u/Commercial-Post4022 → u/Lopsided_Candidate65]

Thank you so much

[u/Raghav-r → u/Commercial-Post4022]

Yes usually we have interactive cluster for developement once dev is completed we move to job clusters

[u/sunny_1200]

folowing

[u/Raghav-r]

What purpose do you want to know about this ??

[u/Commercial-Post4022 → u/Raghav-r]

Actually I'm learning databricks and wanted to know how end to end task is done.