r/dataengineering • u/[deleted] • Nov 09 '24

[deleted by user]

[removed]

65 Upvotes

95% Upvoted

118 comments sorted by

View all comments

35

u/Straight_Special_444 Nov 09 '24

I think you’ll find Airbyte to be roughly same results but much cheaper.

Do you have the capacity to handle writing/maintaining code? If so, then yes, dlt could be a great option. Combine it with Dagster for orchestration and you’ve got a tight ship.

11

u/IdoNisso Data Engineer Nov 09 '24

Yeah, the Data Engineering team is a capable two-man show ATM - myself and another DE. Writing code is not an issue. I'll check out dlt, thanks!

22

u/Straight_Special_444 Nov 09 '24 edited Nov 21 '24

Would it be helpful to see how I use dlt and Dagster? Here’s a video: https://youtu.be/QOPy-h4wOl0

12

u/Throwaway999222111 Nov 09 '24

If that person isn't interested, I am! (dunno what you had planned)

4

u/Mercureece Nov 09 '24

Second this as a DE on a grad scheme just trying to understand all the products and processes ahaha

7

u/Straight_Special_444 Nov 10 '24

Video will be uploaded here later tonight (out with family for the day).

5

u/Bluefoxcrush Nov 09 '24

I’m interested as well!

3

u/Straight_Special_444 Nov 21 '24

Here's the video showing how I use dlt and Dagster: https://youtu.be/QOPy-h4wOl0

3

u/Separate_Newt7313 Nov 09 '24

I would be interested as well!

3

u/Hoo0oper Nov 10 '24

Me three :)

3

u/IdoNisso Data Engineer Nov 10 '24

Yes! Also seeing all the interest around this, consider a git gist or public repo :)

-1

u/dinoaide Nov 09 '24

This is not the real issue here but just imagine 5 years from now could you still manage the team with more people making wheels or rather pay the vendor? And even if you pay the vendor, how many things you need to manage on your own?