Skip to content

Commit 15aed80

Browse files
committed
Add a short post embedding my talk at Data and AI Summit
1 parent 917cd9f commit 15aed80

File tree

1 file changed

+46
-0
lines changed

1 file changed

+46
-0
lines changed
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
layout: post
3+
title: "Presenting Rust and Python Support for Delta Lake"
4+
tags:
5+
- deltalake
6+
- databricks
7+
- featured
8+
- rust
9+
author: rtyler
10+
team: Core Platform
11+
---
12+
13+
Delta Lake is integral to our data platform which is why we have invested
14+
heavily in [delta-rs](https://github.com/delta-io/delta-rs) to support our
15+
non-JVM Delta Lake needs. This year I had the opportunity to share the progress
16+
of delta-rs at Data and AI Summit. Delta-rs was originally started by my colleague [QP](https://github.com/houqp) just over a year ago and it has now grown to now a multi-company project with numerous contributors, and downstream projects such as [kafka-delta-ingest](/blog/2021/kafka-delta-ingest.html).
17+
18+
19+
20+
In the session embedded below, I introduce the delta-rs project which is
21+
helping bring the power of Delta Lake outside of the Spark ecosystem. By
22+
providing a foundational Delta Lake library in Rust, delta-rs can enable native
23+
bindings in Python, Ruby, Golang, and more.We will review what functionality
24+
delta-rs supports in its current Rust and Python APIs and the upcoming roadmap.
25+
26+
I also try to give an overview of one of the first projects to use it in
27+
production:
28+
[kafka-delta-ingest](https://github.com/delta-io/kafka-delta-ingest), which
29+
builds on delta-rs to provide a high throughput service to bring data from
30+
Kafka into Delta Lake.
31+
32+
33+
<center>
34+
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/scYz12UK-OY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
35+
</center>
36+
37+
38+
Investing in data platform tools and automation is a key part of the vision for
39+
Platform Engineering which encompasses Data Engineering, Data Operations, and
40+
Core Platform. We have a [number of open positions](/careers/#open-positions)
41+
at the moment including a position to work closely with me as [Data Engineering
42+
Manager](https://jobs.lever.co/scribd/e2187c1c-a1d6-4b77-bde6-acc997f68156).
43+
The leader of the Data Engineering team will help deliver data tools and
44+
solutions for internal customers building on top of Delta Lake, Databricks,
45+
Airflow, and Kafka. Suffice it to say, there's a lot of really interesting work
46+
to be done!

0 commit comments

Comments
 (0)