"PN Tac2 Training" By CookiePPP (https://pastebin.com/u/CookiePPP) URL: https://pastebin.com/Ddr6D1EF Created on: Wednesday 30th of September 2020 10:18:33 PM CDT Last edit on: Wednesday 30th of September 2020 10:19:32 PM CDT Retrieved on: Saturday 31 of October 2020 11:08:46 PM UTC (base) cookie@pop-os:/media/cookie/Samsung PM961/TensorflowTTS$ CUDA_VISIBLE_DEVICES=3 python3 examples/tacotron2/train_tacotron2.py --train-dir ../dump/train/ --dev-dir ../dump/valid/ --outdir ./tac2out --config ./examples/tacotron2/conf/tacotron2.v1_ms_me48.yaml --use-norm 1 --mixed_precision 0 2020-10-01 04:13:30.506148: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-10-01 04:13:32.143801: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1 2020-10-01 04:13:32.186689: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 04:13:32.188430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:43:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1 coreClock: 1.683GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s 2020-10-01 04:13:32.188451: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-10-01 04:13:32.189814: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2020-10-01 04:13:32.191233: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10 2020-10-01 04:13:32.191411: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10 2020-10-01 04:13:32.192743: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10 2020-10-01 04:13:32.193424: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10 2020-10-01 04:13:32.197289: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2020-10-01 04:13:32.197484: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 04:13:32.200285: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 04:13:32.201843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2020-10-01 04:13:33.871710: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-10-01 04:13:33.896915: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3492995000 Hz 2020-10-01 04:13:33.898465: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55eeb1847680 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-10-01 04:13:33.898489: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-10-01 04:13:34.027573: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 04:13:34.028418: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55eeb1408110 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-10-01 04:13:34.028445: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1 2020-10-01 04:13:34.028688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 04:13:34.029902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:43:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1 coreClock: 1.683GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s 2020-10-01 04:13:34.029943: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-10-01 04:13:34.029980: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2020-10-01 04:13:34.030001: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10 2020-10-01 04:13:34.030021: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10 2020-10-01 04:13:34.030043: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10 2020-10-01 04:13:34.030063: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10 2020-10-01 04:13:34.030085: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2020-10-01 04:13:34.030165: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 04:13:34.031605: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 04:13:34.033076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0 2020-10-01 04:13:34.033117: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-10-01 04:13:34.404235: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-10-01 04:13:34.404281: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263] 0 2020-10-01 04:13:34.404287: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0: N 2020-10-01 04:13:34.404536: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 04:13:34.405327: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-10-01 04:13:34.406044: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9986 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:43:00.0, compute capability: 6.1) 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: hop_size = 600 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: format = npy 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: model_type = tacotron2 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: tacotron2_params = {'dataset': 'ljspeech', 'embedding_hidden_size': 512, 'initializer_range': 0.02, 'embedding_dropout_prob': 0.1, 'n_speakers': 136, 'n_emotions': 22, 'n_conv_encoder': 5, 'encoder_conv_filters': 512, 'encoder_conv_kernel_sizes': 5, 'encoder_conv_activation': 'relu', 'encoder_conv_dropout_rate': 0.5, 'encoder_lstm_units': 256, 'n_prenet_layers': 2, 'prenet_units': 256, 'prenet_activation': 'relu', 'prenet_dropout_rate': 0.5, 'n_lstm_decoder': 1, 'reduction_factor': 1, 'decoder_lstm_units': 1024, 'attention_dim': 128, 'attention_filters': 32, 'attention_kernel': 31, 'n_mels': 80, 'n_conv_postnet': 5, 'postnet_conv_filters': 512, 'postnet_conv_kernel_sizes': 5, 'postnet_dropout_rate': 0.1, 'attention_type': 'lsa'} 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: batch_size = 32 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: remove_short_samples = True 2020-10-01 04:13:47,275 (train_tacotron2:398) INFO: allow_cache = True 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: mel_length_threshold = 32 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: is_shuffle = True 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: use_fixed_shapes = True 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: optimizer_params = {'initial_learning_rate': 0.001, 'end_learning_rate': 1e-05, 'decay_steps': 150000, 'warmup_proportion': 0.02, 'weight_decay': 0.001} 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: train_max_steps = 200000 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: save_interval_steps = 2000 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: eval_interval_steps = 500 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: log_interval_steps = 200 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: start_schedule_teacher_forcing = 200001 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: start_ratio_value = 0.5 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: schedule_decay_steps = 50000 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: end_ratio_value = 0.0 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: num_save_intermediate_results = 1 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: train_dir = ../dump/train/ 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: dev_dir = ../dump/valid/ 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: use_norm = True 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: outdir = ./tac2out 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: config = ./examples/tacotron2/conf/tacotron2.v1_ms_me48.yaml 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: resume = 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: verbose = 1 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: mixed_precision = False 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: pretrained = 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: version = 0.0 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: max_mel_length = 796 2020-10-01 04:13:47,276 (train_tacotron2:398) INFO: max_char_length = 139 2020-10-01 04:13:48.728974: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10 2020-10-01 04:13:48.865129: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 Model: "tacotron2" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= encoder (TFTacotronEncoder) multiple 9432064 _________________________________________________________________ decoder_cell (TFTacotronDeco multiple 18246402 _________________________________________________________________ post_net (TFTacotronPostnet) multiple 5460480 _________________________________________________________________ residual_projection (Dense) multiple 41040 ================================================================= Total params: 33,179,986 Trainable params: 33,169,746 Non-trainable params: 10,240 _________________________________________________________________ [train]: 0%| | 0/200000 [00:00