tf.raw_ops.ExperimentalAutoShardDataset
Creates a dataset that shards the input dataset.
tf.raw_ops.ExperimentalAutoShardDataset(
input_dataset, num_workers, index, output_types, output_shapes,
auto_shard_policy=0, name=None
)
Creates a dataset that shards the input dataset by num_workers, returning a sharded dataset for the index-th worker. This attempts to automatically shard a dataset by examining the Dataset graph and inserting a shard op before the inputs to a reader Dataset (e.g. CSVDataset, TFRecordDataset).
This dataset will throw a NotFound error if we cannot shard the dataset automatically.
| Args | |
|---|---|
input_dataset |
A Tensor of type variant. A variant tensor representing the input dataset. |
num_workers |
A Tensor of type int64. A scalar representing the number of workers to distribute this dataset across. |
index |
A Tensor of type int64. A scalar representing the index of the current worker out of num_workers. |
output_types |
A list of tf.DTypes that has length >= 1. |
output_shapes |
A list of shapes (each a tf.TensorShape or list of ints) that has length >= 1. |
auto_shard_policy |
An optional int. Defaults to 0. |
name |
A name for the operation (optional). |
| Returns | |
|---|---|
A Tensor of type variant. |
© 2020 The TensorFlow Authors. All rights reserved.
Licensed under the Creative Commons Attribution License 3.0.
Code samples licensed under the Apache 2.0 License.
https://www.tensorflow.org/versions/r2.3/api_docs/python/tf/raw_ops/ExperimentalAutoShardDataset