Skip to content

Conversation

@bibhabasumohapatra
Copy link
Contributor

@bibhabasumohapatra bibhabasumohapatra commented Jan 7, 2022

Fixes #2197

Description:

e2 = DeterministicEngine(lambda e, b: None)
print(e2.state_dict())

returns:
AttributeError: 'State' object has no attribute 'rng_states'

Check list:

  • solves the AttributeError: 'State' object has no attribute 'rng_states'
  • add test for the state_dict method

@github-actions github-actions bot added the module: engine Engine module label Jan 7, 2022
@vfdev-5
Copy link
Collaborator

vfdev-5 commented Jan 8, 2022

@bibhabasumohapatra please add a test for state_dict method

def state_dict(self) -> OrderedDict:
setattr(self.state, "rng_states", _get_rng_states())
state_dict = super(DeterministicEngine, self).state_dict()
state_dict["rng_states"] = _get_rng_states()
Copy link
Collaborator

@vfdev-5 vfdev-5 Jan 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep this, instead of setattr(self.state, "rng_states", _get_rng_states())

EDIT:
Main point here is that engine.state.rng_states does not (and is not intended to) reflect rng states at each moment as rng states are dynamically changing. Thus we kept engine.state.rng_states is None once provided rng state was applied in _setup_engine method. If we leave engine.state.rng_states with a certain value then it wont reflect the latest state and thus provides some wrong info. => I propose to keep engine.state.rng_states as None and do not compare with sd["rng_states"].

Copy link
Collaborator

@vfdev-5 vfdev-5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank for the PR @bibhabasumohapatra !

@vfdev-5 vfdev-5 merged commit 192d721 into pytorch:master Jan 10, 2022
@bibhabasumohapatra bibhabasumohapatra deleted the solves-error-rng_states branch January 10, 2022 13:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module: engine Engine module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Load checkpoints within a DeterministicEngine

2 participants