Deploy paws_data_pipeline

[bbucior]

Deployment specifications

• Repository: https://github.com/CodeForPhilly/paws-data-pipeline
• Branch: master
• Project Profile: https://codeforphilly.org/projects/paws_data_pipeline

Additional notes

We are looking to set up a data lake in collaboration with PAWS to manage their data (e.g. volunteering, adoptions, donations). We are still exploring different options for scheduling the pipeline (cron, airflow, etc.) and data storage (e.g. PostgreSQL). What resources could Linode help us with for compute resources and/or storage?

[bbucior]

(also pinging @dsalorio)

[themightychris]

@bbucior Linode is sponsoring Code for Philly with raw compute/storage, and we are administering a community kubernetes cluster on top of that with a volunteer ops team projects and partners can share for support. We still have a lot to explore and evolve in how Code for Philly will interface with projects and manage the cluster.

Our MVP is what you just experienced: open a pull request showing us your docker-compose file and we'll translate it to kubernetes resources we can deploy for you. If you have more advanced needs though you could work directly with the ops team to develop your own kubernetes resources within your own namespace

[machow]

@themightychris is there an avenue for projects to prototype on Linode (using CfP resources)?

For example, DataCamp's chief data scientist set up our entire data pipeline on an amazon ECS instance, and just shelled in to install dependencies, etc..

The engineer in me was very terrified, but in the end it allowed us to develop the process very fast, and to tweak it (we pushed changes to pipeline content via circle CI). Once we knew we had the right thing months later, we containerized it. I'm a big fan of keeping things in containers from the get go, but have noticed it often slow down prototyping.

Is it better for projects to prototype on their own resources (e.g. with some thing they can shell in to), or is there a way for them to do this here?

[themightychris]

That's a tough one, you've got a great point that project need space to experiment quickly online. The problem though is it immediately will turn into things being deployed in one-off environments no one knows how to touch or move

The easy answer is that folks can work locally and should containerize from the start because it's good for participation, but in reality many people don't have portable computers that can run docker well and there's value in being able to share access to scratch spaces online. There's a lot of value in being able to give teams VMs to work together on.

But we do need to stop people from putting domains names on their scratch VMs and calling it deployed. There's basically no way for that not to lead in just a year to our ops team looking at that VM on a bill and having no idea how to update/move whatever is running inside it

How about this:

• we add another issue type to this repo for requesting a team dev workspace
• we provision a standard private VM image with docker and maybe some mini-k8s pre-installed to make for easy prototyping for civic cloud
• we lock down networking on the VM, both as a security precaution since people will be running lots of admin/admin code, and to ensure the unmanaged system doesn't enter a production role
  • firewall all connections to whitelist of team member IPs and ops bastion host
  • maybe: also install zerotier and provision a network for the team. ZT makes it easy then for the team to self-grant any computer or mobile device into being able to connect with their box from any network

@lottspot would it make sense to set up an ansible playbook that can turn a directory of workspace definitions with sets of ssh keys into independent Linode VMs?

[themightychris]

For example, DataCamp's chief data scientist set up our entire data pipeline on an amazon ECS instance, and just shelled in to install dependencies, etc..

The opposing consideration we have to balance against quick demos like that is who has agency now to work on, replicate, or maintain that setup? The state we should be rushing to is one where everyone who joins the team has agency to iterate and deploy. One person being able to do it once doesn't make for a project

[machow]

Sorry, just getting to this--

The opposing consideration we have to balance against quick demos like that is who has agency now to work on, replicate, or maintain that setup? The state we should be rushing to is one where everyone who joins the team has agency to iterate and deploy.

I agree, and think your previous post highlights perfectly the tension between a person using a server like a fancy (un-reproducable) version of their laptop, or worse slapping a domain on it--and rapid experimentation.

Thinking about it now, maybe the key is getting people up to speed on a simple approach--like how they can run this airflow + docker compose example. Once they have it running locally w/ docker-compose, it sounds like they'll be in good place to interface on ops (exactly as you described).

I also don't mind helping taking a stab at making some basic modifications to example repo, and going through productionizing, since it's becoming a common data science workflow.