Finalize full reindex procedure

[yuanzhou]

• Update the step-by-step guide in both README.md and the script help instructions
• Remove misleading print outputs at the end of each command run
• Add a new command to delete those flushYYYYMMDD_fresh_index_* indices AFTER catch-up

Edited on 11/09/2024, catch-up threw this error on PROD

[2024-11-09 12:55:42] INFO in hubmap_translator:1362: Start executing _call_indexer() on uuid: 12dcb9de1b1aa098fabaf7991fd3cf78, entity_type: Upload
[2024-11-09 12:55:42] INFO in hubmap_translator:632: Start executing direct 'entities' updates for entity['uuid']=12dcb9de1b1aa098fabaf7991fd3cf78, entity['entity_type']=Upload
[2024-11-09 12:55:42] INFO in hubmap_translator:1567: Start executing _generate_doc() for Upload of uuid: 12dcb9de1b1aa098fabaf7991fd3cf78 for the entities index group.
[2024-11-09 12:55:42] ERROR in hubmap_translator:1754: Exceptions during executing hubmap_translator._generate_doc()
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 1691, in _generate_doc
    entity['ancestors'] = self._relatives_for_index_group(  relative_ids=ancestor_ids
UnboundLocalError: local variable 'ancestor_ids' referenced before assignment
[2024-11-09 12:55:42] ERROR in hubmap_translator:654: Exception document generation for uuid: 12dcb9de1b1aa098fabaf7991fd3cf78, entity_type: Upload for 'entities' reindex caused 'local variable 'ancestor_ids' referenced before assignment'
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 1691, in _generate_doc
    entity['ancestors'] = self._relatives_for_index_group(  relative_ids=ancestor_ids
UnboundLocalError: local variable 'ancestor_ids' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 641, in _transform_and_write_entity_to_index_group
    private_doc = self._generate_doc(   entity=doc_entity
  File "/usr/src/app/src/hubmap_translator.py", line 1757, in _generate_doc
    raise Exception(e)
Exception: local variable 'ancestor_ids' referenced before assignment
[2024-11-09 12:55:42] ERROR in hubmap_translator:657: For Upload 12dcb9de1b1aa098fabaf7991fd3cf78, failed to generate document for consortium indices.
[2024-11-09 12:55:42] INFO in hubmap_translator:667: Unable to find 'entities' transformer, indexing documents untransformed.
[2024-11-09 12:55:42] ERROR in hubmap_translator:1375: Encountered exception e=local variable 'private_doc' referenced before assignment executing _call_indexer() with uuid: 12dcb9de1b1aa098fabaf7991fd3cf78, entity_type: Upload
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 1367, in _call_indexer
    self._transform_and_write_entity_to_index_group(entity=entity
  File "/usr/src/app/src/hubmap_translator.py", line 668, in _transform_and_write_entity_to_index_group
    docs_to_write_dict[self.index_group_es_indices[index_group]['private']] = private_doc
UnboundLocalError: local variable 'private_doc' referenced before assignment
[2024-11-09 12:55:42] INFO in hubmap_translator:1195: Finished executing reindex_entity() on uuid: 12dcb9de1b1aa098fabaf7991fd3cf78

[yuanzhou]

1/9/2025

Full reindex error:

[2025-01-09 07:40:02] ERROR in hubmap_translator:648: Exception document generation for uuid: e503546b186634bdf46d468a82ee2627, entity_type: Sample for 'portal' reindex caused '502 Server Error: Bad Gateway for url: http://entity-api:8080/children/e503546b186634bdf46d468a82ee2627?property=uuid'
[2025-01-09 07:40:02] ERROR in hubmap_translator:1369: Encountered exception e=local variable 'private_doc' referenced before assignment executing _call_indexer() with uuid: d7b5bbb0fb6031985df6425bd20bee92, entity_type: Sample
[2025-01-09 07:40:02] ERROR in hubmap_translator:1739: Exceptions during executing hubmap_translator._generate_doc()

go-live warning:

[2025-01-09 12:44:55] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): raw.githubusercontent.com:443
/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py:1063: InsecureRequestWarning: Unverified HTTPS request is being made to host 'raw.githubusercontent.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(

catch-up error

[2025-01-09 12:45:58] INFO in hubmap_translator:661: Unable to find 'entities' transformer, indexing documents untransformed.
[2025-01-09 12:45:58] ERROR in hubmap_translator:1369: Encountered exception e=local variable 'private_doc' referenced before assignment executing _call_indexer() with uuid: b1ed06d9475673e6e41c324b0a998eab, entity_type: Upload
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 1361, in _call_indexer
    self._transform_and_write_entity_to_index_group(entity=entity
  File "/usr/src/app/src/hubmap_translator.py", line 662, in _transform_and_write_entity_to_index_group
    docs_to_write_dict[self.index_group_es_indices[index_group]['private']] = private_doc
UnboundLocalError: local variable 'private_doc' referenced before assignment

Full-reindex-go-live-1-9-2025.log
Full-reindex-catch-up-1-9-2025.log

[yuanzhou]

3/5/2025 PROD go-live

[2025-03-05 13:18:45] INFO in fresh_indices:318: ############# Swapping index names to go live via script started at 13:18:45 #############
[2025-03-05 13:18:45] INFO in fresh_indices:320: Swapping index names for create_fill.
[2025-03-05 13:18:45] INFO in fresh_indices:327: Plan to swap hm_prod_public_entities to become flush20250304_fresh_index_hm_prod_public_entities and swap fill20250304_fresh_index_hm_prod_public_entities to become hm_prod_public_entities
[2025-03-05 13:18:45] INFO in fresh_indices:327: Plan to swap hm_prod_consortium_entities to become flush20250304_fresh_index_hm_prod_consortium_entities and swap fill20250304_fresh_index_hm_prod_consortium_entities to become hm_prod_consortium_entities
[2025-03-05 13:18:45] INFO in fresh_indices:327: Plan to swap hm_prod_public_portal to become flush20250304_fresh_index_hm_prod_public_portal and swap fill20250304_fresh_index_hm_prod_public_portal to become hm_prod_public_portal
[2025-03-05 13:18:45] INFO in fresh_indices:327: Plan to swap hm_prod_consortium_portal to become flush20250304_fresh_index_hm_prod_consortium_portal and swap fill20250304_fresh_index_hm_prod_consortium_portal to become hm_prod_consortium_portal
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /hm_prod_public_entities HTTP/1.1" 200 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /flush20250304_fresh_index_hm_prod_public_entities HTTP/1.1" 404 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /fill20250304_fresh_index_hm_prod_public_entities HTTP/1.1" 200 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /hm_prod_consortium_entities HTTP/1.1" 200 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /flush20250304_fresh_index_hm_prod_consortium_entities HTTP/1.1" 404 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /fill20250304_fresh_index_hm_prod_consortium_entities HTTP/1.1" 200 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /hm_prod_public_portal HTTP/1.1" 200 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /flush20250304_fresh_index_hm_prod_public_portal HTTP/1.1" 404 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /fill20250304_fresh_index_hm_prod_public_portal HTTP/1.1" 200 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /hm_prod_consortium_portal HTTP/1.1" 200 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /flush20250304_fresh_index_hm_prod_consortium_portal HTTP/1.1" 404 0
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "HEAD /fill20250304_fresh_index_hm_prod_consortium_portal HTTP/1.1" 200 0
[2025-03-05 13:18:45] DEBUG in fresh_indices:353: Set write block on source_index=hm_prod_public_entities.
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_public_entities/_block/write HTTP/1.1" 200 109
[2025-03-05 13:18:45] INFO in es_manager:236: Set 'write' block on index: hm_prod_public_entities
[2025-03-05 13:18:45] DEBUG in fresh_indices:356: Set write block on destination_index=fill20250304_fresh_index_hm_prod_public_entities.
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_public_entities/_block/write HTTP/1.1" 200 129
[2025-03-05 13:18:45] INFO in es_manager:236: Set 'write' block on index: fill20250304_fresh_index_hm_prod_public_entities
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:45] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/hm_prod_public_entities?wait_for_status=green&timeout=30s HTTP/1.1" 200 263
[2025-03-05 13:18:45] DEBUG in es_manager:291: Wait for index: hm_prod_public_entities to reach "green" health complete.
[2025-03-05 13:18:45] DEBUG in fresh_indices:362: Health of source_index=hm_prod_public_entities is green.
[2025-03-05 13:18:45] DEBUG in fresh_indices:365: Clone source_index=hm_prod_public_entities to flush_index=flush20250304_fresh_index_hm_prod_public_entities.
[2025-03-05 13:18:45] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:46] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_public_entities/_clone/flush20250304_fresh_index_hm_prod_public_entities HTTP/1.1" 200 109
[2025-03-05 13:18:46] INFO in es_manager:261: Cloned source index: hm_prod_public_entities to target index: flush20250304_fresh_index_hm_prod_public_entities
[2025-03-05 13:18:46] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:48] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/flush20250304_fresh_index_hm_prod_public_entities?wait_for_status=green&timeout=30s HTTP/1.1" 200 265
[2025-03-05 13:18:48] DEBUG in es_manager:291: Wait for index: flush20250304_fresh_index_hm_prod_public_entities to reach "green" health complete.
[2025-03-05 13:18:48] DEBUG in fresh_indices:372: Health of flush_index=flush20250304_fresh_index_hm_prod_public_entities is green.
[2025-03-05 13:18:48] DEBUG in fresh_indices:373: Set none block on source_index=hm_prod_public_entities.
[2025-03-05 13:18:48] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:48] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_public_entities/_settings HTTP/1.1" 200 47
[2025-03-05 13:18:48] INFO in es_manager:236: Set 'none' block on index: hm_prod_public_entities
[2025-03-05 13:18:48] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:49] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "DELETE /hm_prod_public_entities HTTP/1.1" 200 47
[2025-03-05 13:18:49] INFO in es_manager:144: Deleted index: hm_prod_public_entities
[2025-03-05 13:18:49] DEBUG in fresh_indices:377: Deleted source_index=hm_prod_public_entities.
[2025-03-05 13:18:49] DEBUG in fresh_indices:382: Clone destination_index=fill20250304_fresh_index_hm_prod_public_entities to source_index=hm_prod_public_entities.
[2025-03-05 13:18:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:49] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_public_entities/_clone/hm_prod_public_entities HTTP/1.1" 200 90
[2025-03-05 13:18:49] INFO in es_manager:261: Cloned source index: fill20250304_fresh_index_hm_prod_public_entities to target index: hm_prod_public_entities
[2025-03-05 13:18:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:52] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/hm_prod_public_entities?wait_for_status=green&timeout=30s HTTP/1.1" 200 265
[2025-03-05 13:18:52] DEBUG in es_manager:291: Wait for index: hm_prod_public_entities to reach "green" health complete.
[2025-03-05 13:18:52] DEBUG in fresh_indices:389: Health of source_index=hm_prod_public_entities is green.
[2025-03-05 13:18:52] DEBUG in fresh_indices:390: Set none block on destination_index=fill20250304_fresh_index_hm_prod_public_entities.
[2025-03-05 13:18:52] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:52] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_public_entities/_settings HTTP/1.1" 200 47
[2025-03-05 13:18:52] INFO in es_manager:236: Set 'none' block on index: fill20250304_fresh_index_hm_prod_public_entities
[2025-03-05 13:18:52] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:52] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "DELETE /fill20250304_fresh_index_hm_prod_public_entities HTTP/1.1" 200 47
[2025-03-05 13:18:52] INFO in es_manager:144: Deleted index: fill20250304_fresh_index_hm_prod_public_entities
[2025-03-05 13:18:52] DEBUG in fresh_indices:394: Deleted destination_index=fill20250304_fresh_index_hm_prod_public_entities.
[2025-03-05 13:18:52] DEBUG in fresh_indices:399: Set none block on source_index=hm_prod_public_entities.
[2025-03-05 13:18:52] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:52] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_public_entities/_settings HTTP/1.1" 200 47
[2025-03-05 13:18:52] INFO in es_manager:236: Set 'none' block on index: hm_prod_public_entities
[2025-03-05 13:18:52] DEBUG in fresh_indices:402: Set none block on flush_index=flush20250304_fresh_index_hm_prod_public_entities.
[2025-03-05 13:18:52] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:52] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /flush20250304_fresh_index_hm_prod_public_entities/_settings HTTP/1.1" 200 47
[2025-03-05 13:18:52] INFO in es_manager:236: Set 'none' block on index: flush20250304_fresh_index_hm_prod_public_entities
[2025-03-05 13:18:52] DEBUG in fresh_indices:353: Set write block on source_index=hm_prod_consortium_entities.
[2025-03-05 13:18:52] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:52] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_consortium_entities/_block/write HTTP/1.1" 200 111
[2025-03-05 13:18:52] INFO in es_manager:236: Set 'write' block on index: hm_prod_consortium_entities
[2025-03-05 13:18:52] DEBUG in fresh_indices:356: Set write block on destination_index=fill20250304_fresh_index_hm_prod_consortium_entities.
[2025-03-05 13:18:52] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:52] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_consortium_entities/_block/write HTTP/1.1" 200 132
[2025-03-05 13:18:52] INFO in es_manager:236: Set 'write' block on index: fill20250304_fresh_index_hm_prod_consortium_entities
[2025-03-05 13:18:52] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:52] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/hm_prod_consortium_entities?wait_for_status=green&timeout=30s HTTP/1.1" 200 263
[2025-03-05 13:18:52] DEBUG in es_manager:291: Wait for index: hm_prod_consortium_entities to reach "green" health complete.
[2025-03-05 13:18:52] DEBUG in fresh_indices:362: Health of source_index=hm_prod_consortium_entities is green.
[2025-03-05 13:18:52] DEBUG in fresh_indices:365: Clone source_index=hm_prod_consortium_entities to flush_index=flush20250304_fresh_index_hm_prod_consortium_entities.
[2025-03-05 13:18:52] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:18:53] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_consortium_entities/_clone/flush20250304_fresh_index_hm_prod_consortium_entities HTTP/1.1" 200 111
[2025-03-05 13:18:53] INFO in es_manager:261: Cloned source index: hm_prod_consortium_entities to target index: flush20250304_fresh_index_hm_prod_consortium_entities
[2025-03-05 13:18:53] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:05] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/flush20250304_fresh_index_hm_prod_consortium_entities?wait_for_status=green&timeout=30s HTTP/1.1" 200 265
[2025-03-05 13:19:05] DEBUG in es_manager:291: Wait for index: flush20250304_fresh_index_hm_prod_consortium_entities to reach "green" health complete.
[2025-03-05 13:19:05] DEBUG in fresh_indices:372: Health of flush_index=flush20250304_fresh_index_hm_prod_consortium_entities is green.
[2025-03-05 13:19:05] DEBUG in fresh_indices:373: Set none block on source_index=hm_prod_consortium_entities.
[2025-03-05 13:19:05] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:05] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_consortium_entities/_settings HTTP/1.1" 200 47
[2025-03-05 13:19:05] INFO in es_manager:236: Set 'none' block on index: hm_prod_consortium_entities
[2025-03-05 13:19:05] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:05] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "DELETE /hm_prod_consortium_entities HTTP/1.1" 200 47
[2025-03-05 13:19:05] INFO in es_manager:144: Deleted index: hm_prod_consortium_entities
[2025-03-05 13:19:05] DEBUG in fresh_indices:377: Deleted source_index=hm_prod_consortium_entities.
[2025-03-05 13:19:05] DEBUG in fresh_indices:382: Clone destination_index=fill20250304_fresh_index_hm_prod_consortium_entities to source_index=hm_prod_consortium_entities.
[2025-03-05 13:19:05] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:06] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_consortium_entities/_clone/hm_prod_consortium_entities HTTP/1.1" 200 93
[2025-03-05 13:19:06] INFO in es_manager:261: Cloned source index: fill20250304_fresh_index_hm_prod_consortium_entities to target index: hm_prod_consortium_entities
[2025-03-05 13:19:06] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:17] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/hm_prod_consortium_entities?wait_for_status=green&timeout=30s HTTP/1.1" 200 265
[2025-03-05 13:19:17] DEBUG in es_manager:291: Wait for index: hm_prod_consortium_entities to reach "green" health complete.
[2025-03-05 13:19:17] DEBUG in fresh_indices:389: Health of source_index=hm_prod_consortium_entities is green.
[2025-03-05 13:19:17] DEBUG in fresh_indices:390: Set none block on destination_index=fill20250304_fresh_index_hm_prod_consortium_entities.
[2025-03-05 13:19:17] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:17] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_consortium_entities/_settings HTTP/1.1" 200 47
[2025-03-05 13:19:17] INFO in es_manager:236: Set 'none' block on index: fill20250304_fresh_index_hm_prod_consortium_entities
[2025-03-05 13:19:17] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:17] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "DELETE /fill20250304_fresh_index_hm_prod_consortium_entities HTTP/1.1" 200 47
[2025-03-05 13:19:17] INFO in es_manager:144: Deleted index: fill20250304_fresh_index_hm_prod_consortium_entities
[2025-03-05 13:19:17] DEBUG in fresh_indices:394: Deleted destination_index=fill20250304_fresh_index_hm_prod_consortium_entities.
[2025-03-05 13:19:17] DEBUG in fresh_indices:399: Set none block on source_index=hm_prod_consortium_entities.
[2025-03-05 13:19:17] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:17] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_consortium_entities/_settings HTTP/1.1" 200 47
[2025-03-05 13:19:17] INFO in es_manager:236: Set 'none' block on index: hm_prod_consortium_entities
[2025-03-05 13:19:17] DEBUG in fresh_indices:402: Set none block on flush_index=flush20250304_fresh_index_hm_prod_consortium_entities.
[2025-03-05 13:19:17] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:17] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /flush20250304_fresh_index_hm_prod_consortium_entities/_settings HTTP/1.1" 200 47
[2025-03-05 13:19:17] INFO in es_manager:236: Set 'none' block on index: flush20250304_fresh_index_hm_prod_consortium_entities
[2025-03-05 13:19:17] DEBUG in fresh_indices:353: Set write block on source_index=hm_prod_public_portal.
[2025-03-05 13:19:17] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:17] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_public_portal/_block/write HTTP/1.1" 200 107
[2025-03-05 13:19:17] INFO in es_manager:236: Set 'write' block on index: hm_prod_public_portal
[2025-03-05 13:19:17] DEBUG in fresh_indices:356: Set write block on destination_index=fill20250304_fresh_index_hm_prod_public_portal.
[2025-03-05 13:19:17] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:17] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_public_portal/_block/write HTTP/1.1" 200 130
[2025-03-05 13:19:17] INFO in es_manager:236: Set 'write' block on index: fill20250304_fresh_index_hm_prod_public_portal
[2025-03-05 13:19:17] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:17] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/hm_prod_public_portal?wait_for_status=green&timeout=30s HTTP/1.1" 200 263
[2025-03-05 13:19:17] DEBUG in es_manager:291: Wait for index: hm_prod_public_portal to reach "green" health complete.
[2025-03-05 13:19:17] DEBUG in fresh_indices:362: Health of source_index=hm_prod_public_portal is green.
[2025-03-05 13:19:17] DEBUG in fresh_indices:365: Clone source_index=hm_prod_public_portal to flush_index=flush20250304_fresh_index_hm_prod_public_portal.
[2025-03-05 13:19:17] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:18] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_public_portal/_clone/flush20250304_fresh_index_hm_prod_public_portal HTTP/1.1" 200 108
[2025-03-05 13:19:18] INFO in es_manager:261: Cloned source index: hm_prod_public_portal to target index: flush20250304_fresh_index_hm_prod_public_portal
[2025-03-05 13:19:18] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:26] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/flush20250304_fresh_index_hm_prod_public_portal?wait_for_status=green&timeout=30s HTTP/1.1" 200 265
[2025-03-05 13:19:26] DEBUG in es_manager:291: Wait for index: flush20250304_fresh_index_hm_prod_public_portal to reach "green" health complete.
[2025-03-05 13:19:26] DEBUG in fresh_indices:372: Health of flush_index=flush20250304_fresh_index_hm_prod_public_portal is green.
[2025-03-05 13:19:26] DEBUG in fresh_indices:373: Set none block on source_index=hm_prod_public_portal.
[2025-03-05 13:19:26] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:26] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_public_portal/_settings HTTP/1.1" 200 47
[2025-03-05 13:19:26] INFO in es_manager:236: Set 'none' block on index: hm_prod_public_portal
[2025-03-05 13:19:26] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:26] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "DELETE /hm_prod_public_portal HTTP/1.1" 200 47
[2025-03-05 13:19:26] INFO in es_manager:144: Deleted index: hm_prod_public_portal
[2025-03-05 13:19:26] DEBUG in fresh_indices:377: Deleted source_index=hm_prod_public_portal.
[2025-03-05 13:19:26] DEBUG in fresh_indices:382: Clone destination_index=fill20250304_fresh_index_hm_prod_public_portal to source_index=hm_prod_public_portal.
[2025-03-05 13:19:26] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:27] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_public_portal/_clone/hm_prod_public_portal HTTP/1.1" 200 88
[2025-03-05 13:19:27] INFO in es_manager:261: Cloned source index: fill20250304_fresh_index_hm_prod_public_portal to target index: hm_prod_public_portal
[2025-03-05 13:19:27] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:38] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/hm_prod_public_portal?wait_for_status=green&timeout=30s HTTP/1.1" 200 265
[2025-03-05 13:19:38] DEBUG in es_manager:291: Wait for index: hm_prod_public_portal to reach "green" health complete.
[2025-03-05 13:19:38] DEBUG in fresh_indices:389: Health of source_index=hm_prod_public_portal is green.
[2025-03-05 13:19:38] DEBUG in fresh_indices:390: Set none block on destination_index=fill20250304_fresh_index_hm_prod_public_portal.
[2025-03-05 13:19:38] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:38] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_public_portal/_settings HTTP/1.1" 200 47
[2025-03-05 13:19:38] INFO in es_manager:236: Set 'none' block on index: fill20250304_fresh_index_hm_prod_public_portal
[2025-03-05 13:19:38] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:38] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "DELETE /fill20250304_fresh_index_hm_prod_public_portal HTTP/1.1" 200 47
[2025-03-05 13:19:38] INFO in es_manager:144: Deleted index: fill20250304_fresh_index_hm_prod_public_portal
[2025-03-05 13:19:38] DEBUG in fresh_indices:394: Deleted destination_index=fill20250304_fresh_index_hm_prod_public_portal.
[2025-03-05 13:19:38] DEBUG in fresh_indices:399: Set none block on source_index=hm_prod_public_portal.
[2025-03-05 13:19:38] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:38] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_public_portal/_settings HTTP/1.1" 200 47
[2025-03-05 13:19:38] INFO in es_manager:236: Set 'none' block on index: hm_prod_public_portal
[2025-03-05 13:19:38] DEBUG in fresh_indices:402: Set none block on flush_index=flush20250304_fresh_index_hm_prod_public_portal.
[2025-03-05 13:19:38] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:38] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /flush20250304_fresh_index_hm_prod_public_portal/_settings HTTP/1.1" 200 47
[2025-03-05 13:19:38] INFO in es_manager:236: Set 'none' block on index: flush20250304_fresh_index_hm_prod_public_portal
[2025-03-05 13:19:38] DEBUG in fresh_indices:353: Set write block on source_index=hm_prod_consortium_portal.
[2025-03-05 13:19:38] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:38] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_consortium_portal/_block/write HTTP/1.1" 200 109
[2025-03-05 13:19:38] INFO in es_manager:236: Set 'write' block on index: hm_prod_consortium_portal
[2025-03-05 13:19:38] DEBUG in fresh_indices:356: Set write block on destination_index=fill20250304_fresh_index_hm_prod_consortium_portal.
[2025-03-05 13:19:38] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:38] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_consortium_portal/_block/write HTTP/1.1" 200 132
[2025-03-05 13:19:38] INFO in es_manager:236: Set 'write' block on index: fill20250304_fresh_index_hm_prod_consortium_portal
[2025-03-05 13:19:38] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:38] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/hm_prod_consortium_portal?wait_for_status=green&timeout=30s HTTP/1.1" 200 263
[2025-03-05 13:19:38] DEBUG in es_manager:291: Wait for index: hm_prod_consortium_portal to reach "green" health complete.
[2025-03-05 13:19:38] DEBUG in fresh_indices:362: Health of source_index=hm_prod_consortium_portal is green.
[2025-03-05 13:19:38] DEBUG in fresh_indices:365: Clone source_index=hm_prod_consortium_portal to flush_index=flush20250304_fresh_index_hm_prod_consortium_portal.
[2025-03-05 13:19:38] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:19:39] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_consortium_portal/_clone/flush20250304_fresh_index_hm_prod_consortium_portal HTTP/1.1" 200 111
[2025-03-05 13:19:39] INFO in es_manager:261: Cloned source index: hm_prod_consortium_portal to target index: flush20250304_fresh_index_hm_prod_consortium_portal
[2025-03-05 13:19:39] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:20:00] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/flush20250304_fresh_index_hm_prod_consortium_portal?wait_for_status=green&timeout=30s HTTP/1.1" 200 265
[2025-03-05 13:20:00] DEBUG in es_manager:291: Wait for index: flush20250304_fresh_index_hm_prod_consortium_portal to reach "green" health complete.
[2025-03-05 13:20:00] DEBUG in fresh_indices:372: Health of flush_index=flush20250304_fresh_index_hm_prod_consortium_portal is green.
[2025-03-05 13:20:00] DEBUG in fresh_indices:373: Set none block on source_index=hm_prod_consortium_portal.
[2025-03-05 13:20:00] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:20:00] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /hm_prod_consortium_portal/_settings HTTP/1.1" 200 47
[2025-03-05 13:20:00] INFO in es_manager:236: Set 'none' block on index: hm_prod_consortium_portal
[2025-03-05 13:20:00] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:20:00] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "DELETE /hm_prod_consortium_portal HTTP/1.1" 400 232
[2025-03-05 13:20:00] ERROR in es_manager:146: Failed to delete index: hm_prod_consortium_portal in elasticsearch.
[2025-03-05 13:20:00] ERROR in es_manager:147: Error Message: {"error":{"root_cause":[{"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[hm_prod_consortium_portal/PCbAv21yTjuuf0501ONwVw]]. Try again after snapshot finishes or cancel the currently running snapshot."}],"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[hm_prod_consortium_portal/PCbAv21yTjuuf0501ONwVw]]. Try again after snapshot finishes or cancel the currently running snapshot."},"status":400}
[2025-03-05 13:20:00] DEBUG in fresh_indices:377: Deleted source_index=hm_prod_consortium_portal.
[2025-03-05 13:20:00] DEBUG in fresh_indices:382: Clone destination_index=fill20250304_fresh_index_hm_prod_consortium_portal to source_index=hm_prod_consortium_portal.
[2025-03-05 13:20:00] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443
[2025-03-05 13:20:00] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250304_fresh_index_hm_prod_consortium_portal/_clone/hm_prod_consortium_portal HTTP/1.1" 400 189
[2025-03-05 13:20:00] ERROR in es_manager:263: Failed to clone source index: fill20250304_fresh_index_hm_prod_consortium_portal  to target index: hm_prod_consortium_portal.
[2025-03-05 13:20:00] ERROR in es_manager:265: Error Payload: {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [hm_prod_consortium_portal/PCbAv21yTjuuf0501ONwVw] already exists","index_uuid":"PCbAv21yTjuuf0501ONwVw","index":"hm_prod_consortium_portal"}],"type":"resource_already_exists_exception","reason":"index [hm_prod_consortium_portal/PCbAv21yTjuuf0501ONwVw] already exists","index_uuid":"PCbAv21yTjuuf0501ONwVw","index":"hm_prod_consortium_portal"},"status":400}
Traceback (most recent call last):
  File "/usr/src/app/scripts/fresh_indices/fresh_indices.py", line 745, in <module>
    swap_index_names_per_strategy(es_mgr=esmanager
  File "/usr/src/app/scripts/fresh_indices/fresh_indices.py", line 383, in swap_index_names_per_strategy
    es_mgr.clone_index(source_index_name=destination_index
  File "/usr/src/app/scripts/fresh_indices/es_manager.py", line 267, in clone_index
    raise Exception(f"Failed to clone source index: {source_index_name} "
Exception: Failed to clone source index: fill20250304_fresh_index_hm_prod_consortium_portal  to target index: hm_prod_consortium_portal with status_code-400, reason-index [hm_prod_consortium_portal/PCbAv21yTjuuf0501ONwVw] already exists. See logs.

[yuanzhou]

4/23/2025, catch-up on TEST after go-live:

[2025-04-23 03:57:21] INFO in fresh_indices:562: The set of UUIDs for entities to re-index is {'4a2b53b9e91741e137966501292632f7', '85479528b4dcb4b0b4631f439de04c22'}
[2025-04-23 03:57:21] DEBUG in fresh_indices:565:  reindex 4a2b53b9e91741e137966501292632f7
[2025-04-23 03:57:21] INFO in hubmap_translator:1184: Start executing reindex_entity() on uuid: 4a2b53b9e91741e137966501292632f7
[2025-04-23 03:57:21] INFO in hubmap_translator:1871: Start executing call_entity_api() on uuid: 4a2b53b9e91741e137966501292632f7
[2025-04-23 03:57:21] DEBUG in connectionpool:246: Starting new HTTP connection (1): entity-api:8080
[2025-04-23 03:57:21] DEBUG in connectionpool:474: http://entity-api:8080 "GET /documents/4a2b53b9e91741e137966501292632f7 HTTP/1.1" 200 456
[2025-04-23 03:57:21] INFO in hubmap_translator:1901: Finished executing call_entity_api() on uuid: 4a2b53b9e91741e137966501292632f7
[2025-04-23 03:57:21] INFO in hubmap_translator:1356: Start executing _call_indexer() on uuid: 4a2b53b9e91741e137966501292632f7, entity_type: Upload
[2025-04-23 03:57:21] INFO in hubmap_translator:626: Start executing direct 'entities' updates for entity['uuid']=4a2b53b9e91741e137966501292632f7, entity['entity_type']=Upload
[2025-04-23 03:57:21] INFO in hubmap_translator:1552: Start executing _generate_doc() for Upload of uuid: 4a2b53b9e91741e137966501292632f7 for the entities index group.
[2025-04-23 03:57:21] ERROR in hubmap_translator:1739: Exceptions during executing hubmap_translator._generate_doc()
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 1676, in _generate_doc
    entity['ancestors'] = self._relatives_for_index_group(  relative_ids=ancestor_ids
UnboundLocalError: local variable 'ancestor_ids' referenced before assignment
[2025-04-23 03:57:21] ERROR in hubmap_translator:648: Exception document generation for uuid: 4a2b53b9e91741e137966501292632f7, entity_type: Upload for 'entities' reindex caused 'local variable 'ancestor_ids' referenced before assignment'
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 1676, in _generate_doc
    entity['ancestors'] = self._relatives_for_index_group(  relative_ids=ancestor_ids
UnboundLocalError: local variable 'ancestor_ids' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 635, in _transform_and_write_entity_to_index_group
    private_doc = self._generate_doc(   entity=doc_entity
  File "/usr/src/app/src/hubmap_translator.py", line 1742, in _generate_doc
    raise Exception(e)
Exception: local variable 'ancestor_ids' referenced before assignment
[2025-04-23 03:57:21] ERROR in hubmap_translator:651: For Upload 4a2b53b9e91741e137966501292632f7, failed to generate document for consortium indices.
[2025-04-23 03:57:21] INFO in hubmap_translator:661: Unable to find 'entities' transformer, indexing documents untransformed.
[2025-04-23 03:57:21] ERROR in hubmap_translator:1369: Encountered exception e=local variable 'private_doc' referenced before assignment executing _call_indexer() with uuid: 4a2b53b9e91741e137966501292632f7, entity_type: Upload
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 1361, in _call_indexer
    self._transform_and_write_entity_to_index_group(entity=entity
  File "/usr/src/app/src/hubmap_translator.py", line 662, in _transform_and_write_entity_to_index_group
    docs_to_write_dict[self.index_group_es_indices[index_group]['private']] = private_doc
UnboundLocalError: local variable 'private_doc' referenced before assignment
[2025-04-23 03:57:21] INFO in hubmap_translator:1189: Finished executing reindex_entity() on uuid: 4a2b53b9e91741e137966501292632f7

[yuanzhou]

5/5/2025 TEST, go-live failed

[2025-05-05 11:51:49] INFO in fresh_indices:318: ############# Swapping index names to go live via script started at 11:51:49 #############
[2025-05-05 11:51:49] INFO in fresh_indices:320: Swapping index names for create_fill.
[2025-05-05 11:51:49] INFO in fresh_indices:327: Plan to swap hm_test_public_entities to become flush20250504_fresh_index_hm_test_public_entities and swap fill20250504_fresh_index_hm_test_public_entities to become hm_test_public_entities
[2025-05-05 11:51:49] INFO in fresh_indices:327: Plan to swap hm_test_consortium_entities to become flush20250504_fresh_index_hm_test_consortium_entities and swap fill20250504_fresh_index_hm_test_consortium_entities to become hm_test_consortium_entities
[2025-05-05 11:51:49] INFO in fresh_indices:327: Plan to swap hm_test_public_portal to become flush20250504_fresh_index_hm_test_public_portal and swap fill20250504_fresh_index_hm_test_public_portal to become hm_test_public_portal
[2025-05-05 11:51:49] INFO in fresh_indices:327: Plan to swap hm_test_consortium_portal to become flush20250504_fresh_index_hm_test_consortium_portal and swap fill20250504_fresh_index_hm_test_consortium_portal to become hm_test_consortium_portal
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /hm_test_public_entities HTTP/1.1" 200 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /flush20250504_fresh_index_hm_test_public_entities HTTP/1.1" 404 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /fill20250504_fresh_index_hm_test_public_entities HTTP/1.1" 200 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /hm_test_consortium_entities HTTP/1.1" 200 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /flush20250504_fresh_index_hm_test_consortium_entities HTTP/1.1" 404 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /fill20250504_fresh_index_hm_test_consortium_entities HTTP/1.1" 200 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /hm_test_public_portal HTTP/1.1" 200 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /flush20250504_fresh_index_hm_test_public_portal HTTP/1.1" 404 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /fill20250504_fresh_index_hm_test_public_portal HTTP/1.1" 200 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /hm_test_consortium_portal HTTP/1.1" 200 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /flush20250504_fresh_index_hm_test_consortium_portal HTTP/1.1" 404 0
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "HEAD /fill20250504_fresh_index_hm_test_consortium_portal HTTP/1.1" 200 0
[2025-05-05 11:51:49] DEBUG in fresh_indices:353: Set write block on source_index=hm_test_public_entities.
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "PUT /hm_test_public_entities/_block/write HTTP/1.1" 200 109
[2025-05-05 11:51:49] INFO in es_manager:236: Set 'write' block on index: hm_test_public_entities
[2025-05-05 11:51:49] DEBUG in fresh_indices:356: Set write block on destination_index=fill20250504_fresh_index_hm_test_public_entities.
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "PUT /fill20250504_fresh_index_hm_test_public_entities/_block/write HTTP/1.1" 200 128
[2025-05-05 11:51:49] INFO in es_manager:236: Set 'write' block on index: fill20250504_fresh_index_hm_test_public_entities
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:49] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/hm_test_public_entities?wait_for_status=green&timeout=30s HTTP/1.1" 200 263
[2025-05-05 11:51:49] DEBUG in es_manager:291: Wait for index: hm_test_public_entities to reach "green" health complete.
[2025-05-05 11:51:49] DEBUG in fresh_indices:362: Health of source_index=hm_test_public_entities is green.
[2025-05-05 11:51:49] DEBUG in fresh_indices:365: Clone source_index=hm_test_public_entities to flush_index=flush20250504_fresh_index_hm_test_public_entities.
[2025-05-05 11:51:49] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:50] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "PUT /hm_test_public_entities/_clone/flush20250504_fresh_index_hm_test_public_entities HTTP/1.1" 200 108
[2025-05-05 11:51:50] INFO in es_manager:261: Cloned source index: hm_test_public_entities to target index: flush20250504_fresh_index_hm_test_public_entities
[2025-05-05 11:51:50] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:53] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "GET /_cluster/health/flush20250504_fresh_index_hm_test_public_entities?wait_for_status=green&timeout=30s HTTP/1.1" 200 265
[2025-05-05 11:51:53] DEBUG in es_manager:291: Wait for index: flush20250504_fresh_index_hm_test_public_entities to reach "green" health complete.
[2025-05-05 11:51:53] DEBUG in fresh_indices:372: Health of flush_index=flush20250504_fresh_index_hm_test_public_entities is green.
[2025-05-05 11:51:53] DEBUG in fresh_indices:373: Set none block on source_index=hm_test_public_entities.
[2025-05-05 11:51:53] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:53] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "PUT /hm_test_public_entities/_settings HTTP/1.1" 200 47
[2025-05-05 11:51:53] INFO in es_manager:236: Set 'none' block on index: hm_test_public_entities
[2025-05-05 11:51:53] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:53] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "DELETE /hm_test_public_entities HTTP/1.1" 400 231
[2025-05-05 11:51:53] ERROR in es_manager:146: Failed to delete index: hm_test_public_entities in elasticsearch.
[2025-05-05 11:51:53] ERROR in es_manager:147: Error Message: {"error":{"root_cause":[{"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[hm_test_public_entities/Ad_zVPrpTbqFxrvai6YkWw]]. Try again after snapshot finishes or cancel the currently running snapshot."}],"type":"snapshot_in_progress_exception","reason":"Cannot delete indices that are being snapshotted: [[hm_test_public_entities/Ad_zVPrpTbqFxrvai6YkWw]]. Try again after snapshot finishes or cancel the currently running snapshot."},"status":400}
[2025-05-05 11:51:53] DEBUG in fresh_indices:377: Deleted source_index=hm_test_public_entities.
[2025-05-05 11:51:53] DEBUG in fresh_indices:382: Clone destination_index=fill20250504_fresh_index_hm_test_public_entities to source_index=hm_test_public_entities.
[2025-05-05 11:51:53] DEBUG in connectionpool:1021: Starting new HTTPS connection (1): search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443
[2025-05-05 11:51:53] DEBUG in connectionpool:474: https://search-hubmap-dev-test-hfnqv4ylo5ywvc42vwnyptbup4.us-east-1.es.amazonaws.com:443 "PUT /fill20250504_fresh_index_hm_test_public_entities/_clone/hm_test_public_entities HTTP/1.1" 400 188
[2025-05-05 11:51:53] ERROR in es_manager:263: Failed to clone source index: fill20250504_fresh_index_hm_test_public_entities  to target index: hm_test_public_entities.
[2025-05-05 11:51:53] ERROR in es_manager:265: Error Payload: {"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [hm_test_public_entities/Ad_zVPrpTbqFxrvai6YkWw] already exists","index_uuid":"Ad_zVPrpTbqFxrvai6YkWw","index":"hm_test_public_entities"}],"type":"resource_already_exists_exception","reason":"index [hm_test_public_entities/Ad_zVPrpTbqFxrvai6YkWw] already exists","index_uuid":"Ad_zVPrpTbqFxrvai6YkWw","index":"hm_test_public_entities"},"status":400}
Traceback (most recent call last):
  File "/usr/src/app/scripts/fresh_indices/fresh_indices.py", line 745, in <module>
    swap_index_names_per_strategy(es_mgr=esmanager
  File "/usr/src/app/scripts/fresh_indices/fresh_indices.py", line 383, in swap_index_names_per_strategy
    es_mgr.clone_index(source_index_name=destination_index
  File "/usr/src/app/scripts/fresh_indices/es_manager.py", line 267, in clone_index
    raise Exception(f"Failed to clone source index: {source_index_name} "
Exception: Failed to clone source index: fill20250504_fresh_index_hm_test_public_entities  to target index: hm_test_public_entities with status_code-400, reason-index [hm_test_public_entities/Ad_zVPrpTbqFxrvai6YkWw] already exists. See logs.
[root@search-api fresh_indices]# ./fresh_indices.sh go-live
Command to execute: 'go-live'
Output will be in ./exec_info

[yuanzhou]

5/14/2025, PROD catch-up

[2025-05-14 12:23:35] DEBUG in connectionpool:474: http://entity-api:8080 "GET /documents/98fbcbd41c18297a31693065c4cb07c8 HTTP/1.1" 200 19690
[2025-05-14 12:23:35] INFO in hubmap_translator:1901: Finished executing call_entity_api() on uuid: 98fbcbd41c18297a31693065c4cb07c8
[2025-05-14 12:23:35] INFO in hubmap_translator:1356: Start executing _call_indexer() on uuid: 98fbcbd41c18297a31693065c4cb07c8, entity_type: Upload
[2025-05-14 12:23:35] INFO in hubmap_translator:626: Start executing direct 'entities' updates for entity['uuid']=98fbcbd41c18297a31693065c4cb07c8, entity['entity_type']=Upload
[2025-05-14 12:23:35] INFO in hubmap_translator:1552: Start executing _generate_doc() for Upload of uuid: 98fbcbd41c18297a31693065c4cb07c8 for the entities index group.
[2025-05-14 12:23:35] ERROR in hubmap_translator:1739: Exceptions during executing hubmap_translator._generate_doc()
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 1676, in _generate_doc
    entity['ancestors'] = self._relatives_for_index_group(  relative_ids=ancestor_ids
UnboundLocalError: local variable 'ancestor_ids' referenced before assignment
[2025-05-14 12:23:35] ERROR in hubmap_translator:648: Exception document generation for uuid: 98fbcbd41c18297a31693065c4cb07c8, entity_type: Upload for 'entities' reindex caused 'local variable 'ancestor_ids' referenced before assignment'
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 1676, in _generate_doc
    entity['ancestors'] = self._relatives_for_index_group(  relative_ids=ancestor_ids
UnboundLocalError: local variable 'ancestor_ids' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 635, in _transform_and_write_entity_to_index_group
    private_doc = self._generate_doc(   entity=doc_entity
  File "/usr/src/app/src/hubmap_translator.py", line 1742, in _generate_doc
    raise Exception(e)
Exception: local variable 'ancestor_ids' referenced before assignment
[2025-05-14 12:23:35] ERROR in hubmap_translator:651: For Upload 98fbcbd41c18297a31693065c4cb07c8, failed to generate document for consortium indices.
[2025-05-14 12:23:35] INFO in hubmap_translator:661: Unable to find 'entities' transformer, indexing documents untransformed.
[2025-05-14 12:23:35] ERROR in hubmap_translator:1369: Encountered exception e=local variable 'private_doc' referenced before assignment executing _call_indexer() with uuid: 98fbcbd41c18297a31693065c4cb07c8, entity_type: Upload
Traceback (most recent call last):
  File "/usr/src/app/src/hubmap_translator.py", line 1361, in _call_indexer
    self._transform_and_write_entity_to_index_group(entity=entity
  File "/usr/src/app/src/hubmap_translator.py", line 662, in _transform_and_write_entity_to_index_group
    docs_to_write_dict[self.index_group_es_indices[index_group]['private']] = private_doc
UnboundLocalError: local variable 'private_doc' referenced before assignment

[yuanzhou]

5/17/2025, PROD full reindex over the weekend
go-live failed:

[2025-05-18 03:38:51] DEBUG in connectionpool:474: https://search-hubmap-prod-e6jfsykycntlyd7m5khviqfbpi.us-east-1.es.amazonaws.com:443 "PUT /fill20250517_fresh_index_hm_prod_consortium_entities/_clone/hm_prod_consortium_entities HTTP/1.1" 500 193
[2025-05-18 03:38:51] ERROR in es_manager:263: Failed to clone source index: fill20250517_fresh_index_hm_prod_consortium_entities  to target index: hm_prod_consortium_entities.
[2025-05-18 03:38:51] ERROR in es_manager:265: Error Payload: {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"index fill20250517_fresh_index_hm_prod_consortium_entities must be read-only to resize index. use \"index.blocks.write=true\""}],"type":"illegal_state_exception","reason":"index fill20250517_fresh_index_hm_prod_consortium_entities must be read-only to resize index. use \"index.blocks.write=true\""},"status":500}
Traceback (most recent call last):
  File "/usr/src/app/scripts/fresh_indices/fresh_indices.py", line 745, in <module>
    swap_index_names_per_strategy(es_mgr=esmanager
  File "/usr/src/app/scripts/fresh_indices/fresh_indices.py", line 383, in swap_index_names_per_strategy
    es_mgr.clone_index(source_index_name=destination_index
  File "/usr/src/app/scripts/fresh_indices/es_manager.py", line 267, in clone_index
    raise Exception(f"Failed to clone source index: {source_index_name} "
Exception: Failed to clone source index: fill20250517_fresh_index_hm_prod_consortium_entities  to target index: hm_prod_consortium_entities with status_code-500, reason-index fill20250517_fresh_index_hm_prod_consortium_entities must be read-only to resize index. use "index.blocks.write=true". See logs.

Fix: deleted all the fill_ and flush_ indices, full reindex again overnight

[yuanzhou]

5/18/2025, ran another round of full reindex overnight, and all went as expected this time. @kburke for the record

[yuanzhou]

5/28/2025 DEV go-live:

[2025-05-28 12:59:25] ERROR in es_manager:263: Failed to clone source index: fill20250528_fresh_index_hm_dev_consortium_portal  to target index: hm_dev_consortium_portal.
[2025-05-28 12:59:25] ERROR in es_manager:265: Error Payload: {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"index fill20250528_fresh_index_hm_dev_consortium_portal must be read-only to resize index. use \"index.blocks.write=true\""}],"type":"illegal_state_exception","reason":"index fill20250528_fresh_index_hm_dev_consortium_portal must be read-only to resize index. use \"index.blocks.write=true\""},"status":500}
Traceback (most recent call last):
  File "/usr/src/app/scripts/fresh_indices/fresh_indices.py", line 745, in <module>
    swap_index_names_per_strategy(es_mgr=esmanager
  File "/usr/src/app/scripts/fresh_indices/fresh_indices.py", line 383, in swap_index_names_per_strategy
    es_mgr.clone_index(source_index_name=destination_index
  File "/usr/src/app/scripts/fresh_indices/es_manager.py", line 267, in clone_index
    raise Exception(f"Failed to clone source index: {source_index_name} "
Exception: Failed to clone source index: fill20250528_fresh_index_hm_dev_consortium_portal  to target index: hm_dev_consortium_portal with status_code-500, reason-index fill20250528_fresh_index_hm_dev_consortium_portal must be read-only to resize index. use "index.blocks.write=true". See logs.

[yuanzhou]

11/01/2025 Full reindex completed but with some failed ids, and the ids list contains duplicates. We should ensure the failed list only contains unique ids.