feat: Evaluator handshake to trainer #5420
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rsareddy0329
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
rsareddy0329
changed the title
rsareddy0329
changed the title
Member
There was a problem hiding this comment.
Can we also make type info available here (instead of Any) for all available options?
| SAGEMAKER_REGION env var or default region. | ||
| sagemaker_session (Optional[Any]): SageMaker session object. If not provided, a default | ||
| session will be created automatically. | ||
| model (Union[str, Any]): Model for evaluation. Can be: |
Member
There was a problem hiding this comment.
Need to update the documentation, on additional values allowed
|
|
||
| try: | ||
| # Handle BaseTrainer type | ||
| if hasattr(v, '__class__') and v.__class__.__name__ == 'BaseTrainer' or hasattr(v, '_latest_training_job'): |
Member
There was a problem hiding this comment.
With the conditioning of v.__class__.__name__ == 'BaseTrainer' will I able to supply SFTTrainer?
Contributor
Author
There was a problem hiding this comment.
Previous jobs were submitted fine, so might have worked.
I moved the code to model_resolution.py to keep it central in this method _resolve_base_model and updated it to isinstance to be sure.
rsareddy0329
temporarily deployed
to
auto-approve
GitHub Actions
Inactive
mufaddal-rohawala
approved these changes
Dec 17, 2025
jam-jee
reviewed
Dec 17, 2025
| benchmark: _Benchmark | ||
| dataset: Union[str, Any] # Required field, must come before optional fields | ||
| subtasks: Optional[Union[str, List[str]]] = None | ||
| evaluate_base_model: bool = True |
Collaborator
There was a problem hiding this comment.
Lets make evaluate_base_model false for all evaluations.
jam-jee
approved these changes
Dec 17, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
arn:aws:sagemaker:us-west-2:729646638167:training-job/pipelines-ayzeo4u9l04n-EvaluateCustomModel-XO0F6U2FMJarn:aws:sagemaker:us-west-2:729646638167:training-job/pipelines-1ufg9wznju1o-EvaluateCustomModel-o2gwjaKdc2arn:aws:sagemaker:us-west-2:729646638167:training-job/CustomInference-wgvcp7tvxhex-cEW1geWEAABy submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.