After the initial pre-training of DeepSeek-V2, we employ YaRN (Peng et al., 2023) to extend the default context window length from 4K to 128K. YaRN was specifically applied to the decoupled shared key k k? ? as it is responsible for carrying RoPE (Su et al., 2024). For YaRN, we set the scale ? to 40, ? to 1, ? to 32, and the target maximum context length to 160K. Under these settings, we can expect the the model to respond well for a context length of 128K. Slightly diverging from original YaRN, due to our distinct attention mechanism, we adjust the length scaling factor to modulate the attention entropy

2 Dependency management The yarn package manager is used to update third party libraries. The configuration lives in scripts/yarn and there is a wrapper script scripts/yarn-update to upgrade the libraries consists of: # Add a yarn package yarn --cwd scripts/yarn add PACKAGE # Edit the script to copy package to the static folder edit scripts/yarn-update # Run the update script ./scripts/yarn-update # Add files

2 Dependency management The yarn package manager is used to update third party libraries. The configuration lives in scripts/yarn and there is a wrapper script scripts/yarn-update to upgrade the libraries consists of: # Add a yarn package yarn --cwd scripts/yarn add PACKAGE # Edit the script to copy package to the static folder edit scripts/yarn-update # Run the update script ./scripts/yarn-update # Add files

2 Dependency management The yarn package manager is used to update third party libraries. The configuration lives in scripts/yarn and there is a wrapper script scripts/yarn-update to upgrade the libraries consists of: # Add a yarn package yarn --cwd scripts/yarn add PACKAGE # Edit the script to copy package to the static folder edit scripts/yarn-update # Run the update script ./scripts/yarn-update # Add files

2 Dependency management The yarn package manager is used to update third party libraries. The configuration lives in scripts/yarn and there is a wrapper script scripts/yarn-update to upgrade the libraries consists of: # Add a yarn package yarn --cwd scripts/yarn add PACKAGE # Edit the script to copy package to the static folder edit scripts/yarn-update # Run the update script ./scripts/yarn-update # Add files

2 Dependency management The yarn package manager is used to update third party libraries. The configuration lives in scripts/yarn and there is a wrapper script scripts/yarn-update to upgrade the libraries consists of: # Add a yarn package yarn --cwd scripts/yarn add PACKAGE # Edit the script to copy package to the static folder edit scripts/yarn-update # Run the update script ./scripts/yarn-update # Add files

Dependency management The yarn package manager is used to update third party libraries. The configuration lives in scripts/yarn and there is a wrapper script scripts/yarn-update to upgrade the libraries consists of: # Add a yarn package yarn --cwd scripts/yarn add PACKAGE # Edit the script to copy package to the static folder edit scripts/yarn-update # Run the update script ./scripts/yarn-update # Add files

Dependency management The yarn package manager is used to update third party libraries. The configuration lives in scripts/yarn and there is a wrapper script scripts/yarn-update to upgrade the libraries consists of: # Add a yarn package yarn --cwd scripts/yarn add PACKAGE # Edit the script to copy package to the static folder edit scripts/yarn-update # Run the update script ./scripts/yarn-update # Add files

Dependency management The yarn package manager is used to update third party libraries. The configuration lives in scripts/yarn and there is a wrapper script scripts/yarn-update to upgrade the libraries consists of: # Add a yarn package yarn --cwd scripts/yarn add PACKAGE # Edit the script to copy package to the static folder edit scripts/yarn-update # Run the update script ./scripts/yarn-update # Add files

2 Dependency management The yarn package manager is used to update third party libraries. The configuration lives in scripts/yarn and there is a wrapper script scripts/yarn-update to upgrade the libraries consists of: # Add a yarn package yarn --cwd scripts/yarn add PACKAGE # Edit the script to copy package to the static folder edit scripts/yarn-update # Run the update script ./scripts/yarn-update # Add files