reindex queue status endpoints

[shirey]

Create two new status endpoints to track the reindex execution in the new prioritized reindex queue functionality of the /reindex/<entity id> endpoint.

GET /reindex-status

returns a json response with the following attributes and associated real-time values:

• num_reindexing the number of items currently being reindexed
• priority_1_queued the number of items in the queue at priority level 1
• priority_2_queued the number of items in the queue at priority level 2
• priority_3_queued the number of items in the queue at priority level 3
• average_reindex_time the real-time running average of how long it takes to reindex entities

Included only if the optional query parameter ?all-queued=true is included

• queued_entities- a list of all queued entities (not including those executing) in the order that they are queued with each item in the list containing an entry about an entity with the following attributes:
  • uuid- the uuid of the entity
  • hubmap_id - the hubmap_id of the entity
  • priority - the reindexing priority (1 ,2 or 3)
  • queued_timestamp - the timestamp of when the entity was originally placed on the queue
  • priority_updated_timestamp - the timestamp of when the priority of the item was re-prioritiezed, either automatically because it was re-requested at a higher priority or because the priority was changed via the PUT /reindex/<entity id>/priority endpoint

Included only if the optional query parameter ?all-reindexing=true is included:

• reindexing_entities - a list of entities currently being reindexed with each item in the list containing an entry about an entity with the following attributes:
  • uuid- the uuid of the entity
  • hubmap_id - the hubmap_id of the entity
  • priority - the reindexing priority (1 ,2 or 3)
  • reindex_start_timestamp - the timestamp of when reindexing for the entity was started
  • queued_timestamp - the timestamp of when the entity was originally placed on the queue
  • priority_updated_timestamp - the timestamp of when the priority of the item was re-prioritiezed, either automatically because it was re-requested at a higher priority or because the priority was changed via the PUT /reindex/<entity id>/priority endpoint

GET /reindex-status/<entity id>

returns a json response with the following attributes and associated real-time values about the requested entity or a 404 if the entity isn't found in the queue
- uuid- the uuid of the entity
- hubmap_id - the hubmap_id of the entity
- priority - the reindexing priority (1 ,2 or 3)
- queued_timestamp - the timestamp of when the entity was originally placed on the queue
- priority_updated_timestamp - the timestamp of when the priority of the item was re-prioritiezed, either automatically because it was re-requested at a higher priority or because the priority was changed via the PUT /reindex/<entity id>/priority endpoint

• execution_number_in_priority - optional, only if not currently executing reindex- the number of entities ahead of the requested entity in the queue that are the same priority level as the entity
• reindex_start_timestamp - optional, only if currently executing reindex- the timestamp of when reindexing for the entity was started

[DerekFurstPitt]

For live average calculation, use a rolling average. within that rolling average track:

• min entity reindex time
• max entity reindex time
• total number of entities reindexed
• total time up
• (optional) the mode time