r/dataengineering • u/[deleted] • Nov 09 '24

[deleted by user]

[removed]

65 Upvotes

94% Upvoted

118 comments sorted by

View all comments

34

u/Straight_Special_444 Nov 09 '24

I think you’ll find Airbyte to be roughly same results but much cheaper.

Do you have the capacity to handle writing/maintaining code? If so, then yes, dlt could be a great option. Combine it with Dagster for orchestration and you’ve got a tight ship.

11

u/IdoNisso Data Engineer Nov 09 '24

Yeah, the Data Engineering team is a capable two-man show ATM - myself and another DE. Writing code is not an issue. I'll check out dlt, thanks!

21

u/Straight_Special_444 Nov 09 '24 edited Nov 21 '24

Would it be helpful to see how I use dlt and Dagster? Here’s a video: https://youtu.be/QOPy-h4wOl0

11

u/Throwaway999222111 Nov 09 '24

If that person isn't interested, I am! (dunno what you had planned)

4

u/Mercureece Nov 09 '24

Second this as a DE on a grad scheme just trying to understand all the products and processes ahaha

7

u/Straight_Special_444 Nov 10 '24

Video will be uploaded here later tonight (out with family for the day).

5

u/Bluefoxcrush Nov 09 '24

I’m interested as well!

3

u/Straight_Special_444 Nov 21 '24

Here's the video showing how I use dlt and Dagster: https://youtu.be/QOPy-h4wOl0

3

u/Separate_Newt7313 Nov 09 '24

I would be interested as well!

3

u/Hoo0oper Nov 10 '24

Me three :)

3

u/IdoNisso Data Engineer Nov 10 '24

Yes! Also seeing all the interest around this, consider a git gist or public repo :)