|
| 1 | +--- |
| 2 | +layout: post |
| 3 | +title: "Presenting Rust and Python Support for Delta Lake" |
| 4 | +tags: |
| 5 | +- deltalake |
| 6 | +- databricks |
| 7 | +- featured |
| 8 | +- rust |
| 9 | +author: rtyler |
| 10 | +team: Core Platform |
| 11 | +--- |
| 12 | + |
| 13 | +Delta Lake is integral to our data platform which is why we have invested |
| 14 | +heavily in [delta-rs](https://github.com/delta-io/delta-rs) to support our |
| 15 | +non-JVM Delta Lake needs. This year I had the opportunity to share the progress |
| 16 | +of delta-rs at Data and AI Summit. Delta-rs was originally started by my colleague [QP](https://github.com/houqp) just over a year ago and it has now grown to now a multi-company project with numerous contributors, and downstream projects such as [kafka-delta-ingest](/blog/2021/kafka-delta-ingest.html). |
| 17 | + |
| 18 | + |
| 19 | + |
| 20 | +In the session embedded below, I introduce the delta-rs project which is |
| 21 | +helping bring the power of Delta Lake outside of the Spark ecosystem. By |
| 22 | +providing a foundational Delta Lake library in Rust, delta-rs can enable native |
| 23 | +bindings in Python, Ruby, Golang, and more.We will review what functionality |
| 24 | +delta-rs supports in its current Rust and Python APIs and the upcoming roadmap. |
| 25 | + |
| 26 | +I also try to give an overview of one of the first projects to use it in |
| 27 | +production: |
| 28 | +[kafka-delta-ingest](https://github.com/delta-io/kafka-delta-ingest), which |
| 29 | +builds on delta-rs to provide a high throughput service to bring data from |
| 30 | +Kafka into Delta Lake. |
| 31 | + |
| 32 | + |
| 33 | +<center> |
| 34 | +<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/scYz12UK-OY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe> |
| 35 | +</center> |
| 36 | + |
| 37 | + |
| 38 | +Investing in data platform tools and automation is a key part of the vision for |
| 39 | +Platform Engineering which encompasses Data Engineering, Data Operations, and |
| 40 | +Core Platform. We have a [number of open positions](/careers/#open-positions) |
| 41 | +at the moment including a position to work closely with me as [Data Engineering |
| 42 | +Manager](https://jobs.lever.co/scribd/e2187c1c-a1d6-4b77-bde6-acc997f68156). |
| 43 | +The leader of the Data Engineering team will help deliver data tools and |
| 44 | +solutions for internal customers building on top of Delta Lake, Databricks, |
| 45 | +Airflow, and Kafka. Suffice it to say, there's a lot of really interesting work |
| 46 | +to be done! |
0 commit comments