number_examples和batch_size之间有什么关系吗？

Question

老师您好，我在跑这部分的代码的时候，不太清楚number_examples和batch_size之间还有什么影响吗？

我把number_examples设置为12，把batch_size设置为9的时候还一切正常，但是设置为10、11、12的时候均抛出异常，这是为什么？

代码如下（只是对您的代码在pycharm中整理了一下）：

from seq2seq import preprocessing as pre
import tensorflow as tf

data_path = 'data/toydata.txt'
num_examples = 12
input_tensor, target_tensor, inp_lang, targ_lang = pre.load_dataset(data_path, num_examples)
input_tensor_train, _, target_tensor_train, _ = pre.data_split(data_path, num_examples)
BUFFER_SIZE = len(input_tensor_train)
BATCH_SIZE = 9
steps_per_epoch = len(input_tensor_train) // BATCH_SIZE
embedding_dim = 256
units = 1024
vocab_inp_size = len(inp_lang.word_index) + 1
vocab_tar_size = len(targ_lang.word_index) + 1
print(vocab_inp_size, vocab_tar_size)

dataset = tf.data.Dataset.from_tensor_slices((input_tensor_train, target_tensor_train)).shuffle(BUFFER_SIZE)
dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)
example_input_batch, example_target_batch = next(iter(dataset))
print(example_input_batch)

Answer 1

正十七 2021-06-27 15:49:23

num_examples是load进来的样本数目，batch_size是每次训练batch里的样本数，理论上说，这里只要batch_size <= num_examples就够了。

1. 跑出的异常具体是什么啊？

2. num_examples = 12也太小了，我在代码中设置的是30000
   

• 提问者 慕九州3352676 #1

  第一部分：
  Traceback (most recent call last):
    File "/data/zhangxy/condenv/tf20/lib/python3.6/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 666, in next
      return self._next_internal()
    File "/data/zhangxy/condenv/tf20/lib/python3.6/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 651, in _next_internal
      output_shapes=self._flat_output_shapes)
    File "/data/zhangxy/condenv/tf20/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_dataset_ops.py", line 2673, in iterator_get_next_sync
      _six.raise_from(_core._status_to_exception(e.code, message), None)
    File "<string>", line 3, in raise_from
  tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence [Op:IteratorGetNextSync]
  
  During handling of the above exception, another exception occurred:

• 提问者 慕九州3352676 #2

  第二部分：
  Traceback (most recent call last):
    File "/data/zhangxy/hello_pytorch/NMT/main.py", line 23, in <module>
      example_input_batch, example_target_batch = next(iter(dataset))
    File "/data/zhangxy/condenv/tf20/lib/python3.6/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 622, in __next__
      return self.next()
    File "/data/zhangxy/condenv/tf20/lib/python3.6/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 668, in next
      raise StopIteration
  StopIteration

• 提问者 慕九州3352676 #3

  非常感谢！