RandomReplay< EnvironmentType > Class Template Reference

Implementation of random experience replay. More...

Public Types
using	ActionType = typename EnvironmentType::Action
	Convenient typedef for action. More...
using	StateType = typename EnvironmentType::State
	Convenient typedef for state. More...

Public Member Functions
	RandomReplay ()
	RandomReplay (const size_t batchSize, const size_t capacity, const size_t dimension=StateType::dimension)
	Construct an instance of random experience replay class. More...
void	Sample (arma::mat &sampledStates, arma::icolvec &sampledActions, arma::colvec &sampledRewards, arma::mat &sampledNextStates, arma::icolvec &isTerminal)
	Sample some experiences. More...
const size_t &	Size ()
	Get the number of transitions in the memory. More...
void	Store (const StateType &state, ActionType action, double reward, const StateType &nextState, bool isEnd)
	Store the given experience. More...
void	Update (arma::mat, arma::icolvec, arma::mat, arma::mat &)
	Update the priorities of transitions and Update the gradients. More...

Detailed Description

template

<

typename

EnvironmentType

>

class mlpack::rl::RandomReplay< EnvironmentType >

Implementation of random experience replay.

At each time step, interactions between the agent and the environment will be saved to a memory buffer. When necessary, we can simply sample previous experiences from the buffer to train the agent. Typically this would be a random sample and the memory will be a First-In-First-Out buffer.

For more information, see the following.

@phdthesis{lin1993reinforcement,
 title  = {Reinforcement learning for robots using neural networks},
 author = {Lin, Long-Ji},
 year   = {1993},
 school = {Fujitsu Laboratories Ltd}
}

Template Parameters

EnvironmentType

Desired task.

Definition at line 43 of file random_replay.hpp.

Member Typedef Documentation

ActionType

using ActionType = typename EnvironmentType::Action

Convenient typedef for action.

Definition at line 47 of file random_replay.hpp.

StateType

using StateType = typename EnvironmentType::State

Convenient typedef for state.

Definition at line 50 of file random_replay.hpp.

Constructor & Destructor Documentation

RandomReplay() [1/2]

RandomReplay ( )

inline

Definition at line 52 of file random_replay.hpp.

RandomReplay() [2/2]

RandomReplay ( const size_t  batchSize, const size_t  capacity, const size_t  dimension = StateType::dimension  )

inline

Construct an instance of random experience replay class.

Parameters

batchSize	Number of examples returned at each sample.
capacity	Total memory size in terms of number of examples.
dimension	The dimension of an encoded state.

Definition at line 66 of file random_replay.hpp.

Member Function Documentation

Sample()

void Sample ( arma::mat &  sampledStates, arma::icolvec &  sampledActions, arma::colvec &  sampledRewards, arma::mat &  sampledNextStates, arma::icolvec &  isTerminal  )

inline

Sample some experiences.

Parameters

sampledStates	Sampled encoded states.
sampledActions	Sampled actions.
sampledRewards	Sampled rewards.
sampledNextStates	Sampled encoded next states.
isTerminal	Indicate whether corresponding next state is terminal state.

Definition at line 118 of file random_replay.hpp.

Size()

const size_t& Size ( )

inline

Get the number of transitions in the memory.

Returns

Actual used memory size

Definition at line 140 of file random_replay.hpp.

Store()

void Store ( const StateType &  state, ActionType  action, double  reward, const StateType &  nextState, bool  isEnd  )

inline

Store the given experience.

Parameters

state	Given state.
action	Given action.
reward	Given reward.
nextState	Given next state.
isEnd	Whether next state is terminal state.

Definition at line 89 of file random_replay.hpp.

Update()

void Update ( arma::mat  , arma::icolvec  , arma::mat  , arma::mat &    )

inline

Update the priorities of transitions and Update the gradients.

Parameters

target	The learned value
sampledActions	Agent's sampled action
nextActionValues	Agent's next action
gradients	The model's gradients

Definition at line 153 of file random_replay.hpp.

---

The documentation for this class was generated from the following file:

• /home/jenkins-mlpack/mlpack.org/_src/mlpack-3.2.1/src/mlpack/methods/reinforcement_learning/replay/random_replay.hpp