Is there a simple way to set epochs when using TFRecords with Tensorflow EstimatorsTensorFlow - Read all examples from a TFRecords at once?Obtaining total number of records from .tfrecords file in TensorflowHow to inspect a Tensorflow .tfrecord file?Numpy to TFrecords: Is there a more simple way to handle batch inputs from tfrecords?Tensorflow, read tfrecord without a graphTensorflow tfrecord not being read correctlyParsing TFRecord when in eager executionSplitting dataset element into multiple dataset elementstensorflow estimator training only runs half of the stepsHow to fix a 'OutOfRangeError: End of sequence' error when training a CNN with tensorflow?
Was there ever any real use for a 6800-based Apple I?
Increase height of laser cut design file for enclosure
Unit Test - Testing API Methods
Noob at soldering, can anyone explain why my circuit won't work?
Is it a bad idea to replace pull-up resistors with hard pull-ups?
Why use steam instead of just hot air?
Renting a house to a graduate student in my department
What does it mean with the ask price is below the last price?
On studying Computer Science vs. Software Engineering to become a proficient coder
Why in a Ethernet LAN, a packet sniffer can obtain all packets sent over the LAN?
Adding slope values to attribute table (QGIS 3)
Will change of address affect direct deposit?
Why was the Ancient One so hesitant to teach Dr. Strange the art of sorcery?
Why did God specifically target the firstborn in the 10th plague (Exodus 12:29-36)?
We are two immediate neighbors who forged our own powers to form concatenated relationship. Who are we?
Washer drain pipe overflow
On what legal basis did the UK remove the 'European Union' from its passport?
Can the sorting of a list be verified without comparing neighbors?
find not returning expected files
What food production methods would allow a metropolis like New York to become self sufficient
The lexical root of the perfect tense forms differs from the lexical root of the infinitive form
How can I answer high-school writing prompts without sounding weird and fake?
histogram using edges
Would an 8% reduction in drag outweigh the weight addition from this custom CFD-tested winglet?
Is there a simple way to set epochs when using TFRecords with Tensorflow Estimators
TensorFlow - Read all examples from a TFRecords at once?Obtaining total number of records from .tfrecords file in TensorflowHow to inspect a Tensorflow .tfrecord file?Numpy to TFrecords: Is there a more simple way to handle batch inputs from tfrecords?Tensorflow, read tfrecord without a graphTensorflow tfrecord not being read correctlyParsing TFRecord when in eager executionSplitting dataset element into multiple dataset elementstensorflow estimator training only runs half of the stepsHow to fix a 'OutOfRangeError: End of sequence' error when training a CNN with tensorflow?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
There is a nice way to set epochs when feeding numpy arrays into an estimator
tf.estimator.inputs.numpy_input_fn(
x,
y=None,
batch_size=128,
num_epochs=1 ,
shuffle=None,
queue_capacity=1000,
num_threads=1
)
But I can't track down a similar method with TFRecords, most people seem to just stick it in a loop
i = 0
while ( i < 100000):
model.train(input_fn=input_fn, steps=100)
Is there a clean way to explicitly set the number of epochs for TFRecords with estimators ?
tensorflow tfrecord
asked Mar 23 at 10:44
NiallJGNiallJG
1,0601319
add a comment |
There is a nice way to set epochs when feeding numpy arrays into an estimator
tf.estimator.inputs.numpy_input_fn(
x,
y=None,
batch_size=128,
num_epochs=1 ,
shuffle=None,
queue_capacity=1000,
num_threads=1
)
But I can't track down a similar method with TFRecords, most people seem to just stick it in a loop
i = 0
while ( i < 100000):
model.train(input_fn=input_fn, steps=100)
Is there a clean way to explicitly set the number of epochs for TFRecords with estimators ?
tensorflow tfrecord
asked Mar 23 at 10:44
NiallJGNiallJG
1,0601319
are you using dataset api?
– Sharky
Mar 23 at 17:12
add a comment |
There is a nice way to set epochs when feeding numpy arrays into an estimator
tf.estimator.inputs.numpy_input_fn(
x,
y=None,
batch_size=128,
num_epochs=1 ,
shuffle=None,
queue_capacity=1000,
num_threads=1
)
But I can't track down a similar method with TFRecords, most people seem to just stick it in a loop
i = 0
while ( i < 100000):
model.train(input_fn=input_fn, steps=100)
Is there a clean way to explicitly set the number of epochs for TFRecords with estimators ?
tensorflow tfrecord
asked Mar 23 at 10:44
NiallJGNiallJG
1,0601319
There is a nice way to set epochs when feeding numpy arrays into an estimator
tf.estimator.inputs.numpy_input_fn(
x,
y=None,
batch_size=128,
num_epochs=1 ,
shuffle=None,
queue_capacity=1000,
num_threads=1
)
But I can't track down a similar method with TFRecords, most people seem to just stick it in a loop
i = 0
while ( i < 100000):
model.train(input_fn=input_fn, steps=100)
Is there a clean way to explicitly set the number of epochs for TFRecords with estimators ?
tensorflow tfrecord
tensorflow tfrecord
asked Mar 23 at 10:44
NiallJGNiallJG
1,0601319
asked Mar 23 at 10:44
NiallJGNiallJG
1,0601319
asked Mar 23 at 10:44
NiallJGNiallJG
1,0601319
asked Mar 23 at 10:44
NiallJGNiallJG
1,0601319
asked Mar 23 at 10:44
NiallJGNiallJG
1,0601319
1,0601319
are you using dataset api?
– Sharky
Mar 23 at 17:12
add a comment |
are you using dataset api?
– Sharky
Mar 23 at 17:12
are you using dataset api?
– Sharky
Mar 23 at 17:12
are you using dataset api?
– Sharky
Mar 23 at 17:12
add a comment |
1 Answer
1
active
oldest
votes
You can set number of epoch with dataset.repeat(num_epochs). Dataset pipeline outputs a dataset object, a tuple (features, labels) of batch size, that is inputed to model.train()
dataset = tf.data.TFRecordDataset(file.tfrecords)
dataset = tf.shuffle().repeat()
...
dataset = dataset.batch()
In order to make it work, you set model.train(steps=None, max_steps=None) In this case, you let Dataset API to handle epochs count by generating tf.errors.OutOfRange error or StopIteration exception once num_epoch is reached.
answered Mar 23 at 21:20
SharkySharky
2,2512918
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55312887%2fis-there-a-simple-way-to-set-epochs-when-using-tfrecords-with-tensorflow-estimat%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can set number of epoch with dataset.repeat(num_epochs). Dataset pipeline outputs a dataset object, a tuple (features, labels) of batch size, that is inputed to model.train()
dataset = tf.data.TFRecordDataset(file.tfrecords)
dataset = tf.shuffle().repeat()
...
dataset = dataset.batch()
In order to make it work, you set model.train(steps=None, max_steps=None) In this case, you let Dataset API to handle epochs count by generating tf.errors.OutOfRange error or StopIteration exception once num_epoch is reached.
answered Mar 23 at 21:20
SharkySharky
2,2512918
add a comment |
You can set number of epoch with dataset.repeat(num_epochs). Dataset pipeline outputs a dataset object, a tuple (features, labels) of batch size, that is inputed to model.train()
dataset = tf.data.TFRecordDataset(file.tfrecords)
dataset = tf.shuffle().repeat()
...
dataset = dataset.batch()
In order to make it work, you set model.train(steps=None, max_steps=None) In this case, you let Dataset API to handle epochs count by generating tf.errors.OutOfRange error or StopIteration exception once num_epoch is reached.
answered Mar 23 at 21:20
SharkySharky
2,2512918
add a comment |
You can set number of epoch with dataset.repeat(num_epochs). Dataset pipeline outputs a dataset object, a tuple (features, labels) of batch size, that is inputed to model.train()
dataset = tf.data.TFRecordDataset(file.tfrecords)
dataset = tf.shuffle().repeat()
...
dataset = dataset.batch()
In order to make it work, you set model.train(steps=None, max_steps=None) In this case, you let Dataset API to handle epochs count by generating tf.errors.OutOfRange error or StopIteration exception once num_epoch is reached.
answered Mar 23 at 21:20
SharkySharky
2,2512918
You can set number of epoch with dataset.repeat(num_epochs). Dataset pipeline outputs a dataset object, a tuple (features, labels) of batch size, that is inputed to model.train()
dataset = tf.data.TFRecordDataset(file.tfrecords)
dataset = tf.shuffle().repeat()
...
dataset = dataset.batch()
In order to make it work, you set model.train(steps=None, max_steps=None) In this case, you let Dataset API to handle epochs count by generating tf.errors.OutOfRange error or StopIteration exception once num_epoch is reached.
answered Mar 23 at 21:20
SharkySharky
2,2512918
answered Mar 23 at 21:20
SharkySharky
2,2512918
answered Mar 23 at 21:20
SharkySharky
2,2512918
answered Mar 23 at 21:20
SharkySharky
2,2512918
2,2512918
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55312887%2fis-there-a-simple-way-to-set-epochs-when-using-tfrecords-with-tensorflow-estimat%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
are you using dataset api?
– Sharky
Mar 23 at 17:12