also retry index download on exists-in-archive, change locking

[syphar]

I'm sorry for the whole back and forth here.

With the time I have difficulties reproducing the load-issue locally, so it's testing in production :)
( hopefully with little impact).

This sentry error pointed me to

1. I didn't add the "fallback" that protects against some index edge cases to all places where we work with the index. Thats in this PR.
2. I went back to a simpler solution for the download-or-fetch problem, between the old and the new way.

With a warm local cache all should be fine, I have to find a good time when I just test the cold cache.

the whole "cold cache" issue will bite us when we migrate our webserver to ecs, I assume there we'll have a fresh filesystem for every deploy

[GuillaumeGomez]

Thanks!