TODO: In practice, it is much easier to train the model by TensorFlow, and then use the keras.layers.get_weights to output the coefficents of each layers of the trained model. These coefficents can be used directly to inference in RTX Neural Shading (which is based on (Direct3D12 Wave Matrix) or (Vulkan Cooperative Matrix)).
The inference shader source is fixed, and can be merely treated as the regular game engine material asset. And these coefficents can be merely treated as the regular game engine texture assets. And the TensorFlow, which is used to generate these coefficents, can be merely treated as the regular game DCC tool. Perhaps every artist will be able to use the TensorFlow in the future!
Note: the coefficients are composed of both the weights and the bias.
| Type | Description |
|---|---|
| uint32_t | FourCC('N', 'T', 'M', ' ') |
| uint32_t | number of frequencies |
| uint32_t | number of the texture mapping layers |
| uint32_t | number of the coefficients of the 1st texture mapping layer |
| float [ ] | coefficients of the 1st texture mapping layer |
| uint32_t | number of the coefficients of the 2nd texture mapping layer |
| float [ ] | coefficients of the 2nd texture mapping layer |
| ... | ... |
The size of the assets of the MMO can be higher than 100GB. If the size of the neutral texture assets can be less than the regular game texture assets, this technique can be useful. All the texture sampling operations can be replaced by inferencing from the coefficients buffers.
The texture cache is always highly optimized by the GPU (for more information). Since we store the neutral texture assets in the buffers, the cache missing can be much higher. In addition, the mipmap level can not be easily supported.
TODO: If we change the memory layout of the target texture according to the AMD addrlib before we train the model, will the result of the trained model be better?
