upload nightly tps metrics to S3

[THardy98]

What was changed

Upload nightly prometheus worker metrics to S3 for querying in Omni

Why?

observability

[Sushisource]

Cool!

We'll need a plan for how to delete old stuff, too.

We might want up make part of the upload job include cleaning up old stuff. Ofc, we'll also need to decide what we want to keep. We might want to keep every day for the last 3 months, then every week for the last year, plus the metrics for releases or something.

That logic has to live somewhere, though. This might push you to consolidate this uploading logic into a shared GH workflow that can be pointed at by the lang repos.

[VegetarianOrc]

Cool!

We'll need a plan for how to delete old stuff, too.

We might want up make part of the upload job include cleaning up old stuff. Ofc, we'll also need to decide what we want to keep. We might want to keep every day for the last 3 months, then every week for the last year, plus the metrics for releases or something.

That logic has to live somewhere, though. This might push you to consolidate this uploading logic into a shared GH workflow that can be pointed at by the lang repos.

Not sure if it's an option, but it's worth considering using an S3 lifecycle policy to dictate how the files get cleaned up or moved to cheaper infrequent access storage.