Translate to English:

For the environment interaction that generates episodes, keep that code unchanged. Assume this produces episodes labeled ep_0, ep_1, ep_3, ... each with different lengths.
Suppose the training batch size is B. Using the episodes from step 1, sample them with probability proportional to their length and concatenate them to build B rollouts. For example, r_0 = ep_4 + ep_1 + ep_0 ..., r_1 = ep_24 + ep_12 + ep_22. For every rollout we record, at each timestep, whether it is the start of a new episode. This gives an
```
is_first
```
sequence like True, False, False, ..., True for each rollout.
Let the training sequence length be T. Slice r_0, r_1, ..., r_B along the time axis to form chunks of shape (B, T, C, W, H), along with reward, done, and the
```
is_first
```
flags. Each chunk is what we feed to the GPU.
For each rollout, only keep as many steps as needed. When building rollouts, if the current length already exceeds T, stop appending new episodes. After removing a T-step chunk, inspect each rollout again; if its remaining length is below T, sample more episodes and append until its length exceeds T. This data-preparation pipeline can run in parallel with training.
During training, if sequence b in the batch has
```
is_first[b, t] == True
```
, reset the model state h for that sequence at time t. In all other cases, carry h forward to the next timestep, and when a batch ends, retain h because the next batch continues the same sequence in time.
Note that within a batch, different sequences may hit
```
is_first == True
```
at different timesteps. To keep computations vectorized, we may need some tricks here. If you are unsure how to vectorize this, let me know.

For the environment interaction that generates episodes, keep that code unchanged. Assume this produces episodes labeled ep_0, ep_1, ep_3, ... each with different lengths.
Suppose the training batch size is B. Using the episodes from step 1, sample them with probability proportional to their length and concatenate them to build B rollouts. For example, r_0 = ep_4 + ep_1 + ep_0 ..., r_1 = ep_24 + ep_12 + ep_22. For every rollout we record, at each timestep, whether it is the start of a new episode. This gives an
```
is_first
```
sequence like True, False, False, ..., True for each rollout.
Let the training sequence length be T. Slice r_0, r_1, ..., r_B along the time axis to form chunks of shape (B, T, C, W, H), along with reward, done, and the
```
is_first
```
flags. Each chunk is what we feed to the GPU.
For each rollout, only keep as many steps as needed. When building rollouts, if the current length already exceeds T, stop appending new episodes. After removing a T-step chunk, inspect each rollout again; if its remaining length is below T, sample more episodes and append until its length exceeds T. This data-preparation pipeline can run in parallel with training.
During training, if sequence b in the batch has
```
is_first[b, t] == True
```
, reset the model state h for that sequence at time t. In all other cases, carry h forward to the next timestep, and when a batch ends, retain h because the next batch continues the same sequence in time.
Note that within a batch, different sequences may hit
```
is_first == True
```
at different timesteps. To keep computations vectorized, we may need some tricks here. If you are unsure how to vectorize this, let me know.

Translate to English:

Related Skills

Nano Banana Pro

Markdown Converter

1password