Styjun

NOTE: this question has an associated colab and is based on the pseudo-code for tf.contrib.factorization.WALSModel.

In short, the question is how to complete the pseudo-code for WALSModel (see colab)

Tensorflow's documentation is not complete nor necessarily the best. It is not uncommon for a documentation page to simply link to .proto file (which may have some comments and perhaps an example). For example ClusterDef, Example, Feature, JobDef, etc all link to a .proto file for the user to decipher.

This can make the conversion of pseudo-code a lengthy and convoluted process, sometimes taking a user working at a "higher" level part of the api down into the trenches.

Such is the case with tf.contrib.factorization.WALSModel pseudo-code.

Part of the reason the WALSModel pseudo-code is particularly hard to decipher is that the pseudo-code is not for the base use case (feeding a matrix into WALSModel, running for i iterations, and getting the resultant row_factors/col_factors). Rather this pseudo-code aims to demonstrate how to run WALSModel perhaps in a distributed setting with a sharded matrix (which may be required for large input matricies).

However, this pseudo-code fails to provide sufficient guidance for implementing a distributed, sharded WALSModel.

Why?

For starters, the first mention that the input matrix should be shared is not until defining the update operations:

_, row_update_op, unreg_row_loss, row_reg, _ = model.update_row_factors(
 sp_input = matrix_slices_from_queue_for_worker_shard
)

The pseudo-code variable matrix_slices_from_queue_for_worker_shard provides no indication as to how the user should attempt this.

Further examples of large gaps in the pseudo code come just a few lines later:

# model_init_op is passed to Supervisor. Chief trainer runs it. Other
# trainers wait.
sv = tf.train.Supervisor(is_chief=is_chief,
 ...,
 init_op=tf.group(..., model_init_op, ...), ...)

In this large pseudo-code block there is mention of where / how the chief training should be set up or how have the other trainers wait.

It is understandable that this pseudo-code is specifically for the WALSModel and not for queue trainers. However, given that the tf.train documentation links to a non-existent "Training" guide (just lists the modules / classes, many of which link to .proto files), where should one go to learn how to properly implement this?

So I set off to try an fill in this pseudo-code to the best of my ability (see colab and quickly hit a dead end.

I would appreciate any assistance / guidance in completing this pseudo-code (distributed, sharded WALSModel) or where to read more about how to do this.

Of use might be the semi-related issue 26928, where @walidk links to some test code for the tf.Estimator version of WALSModel (WALSMatrixFactorization), which seems to have some more code for sharding, but since it is an Estimator, it is not necessarily clear what is needed just for WALSModel in relation to the pseudo-code provided on the WALSModel documentation page.