Styjun

Starting from the universal-sentence-encoder in TensorFlow.js, I noticed that the range of the numbers in the embeddings wasn't what I expected. I was expecting some distribution between [0-1] or [-1,1] but don't see either of these.

For the sentence "cats are great!" here's a visualization, where each dimension is projected onto a scale from [-0.5, 0.5]:

Here's the same kind of visualization for "i wonder what this sentence's embedding will be" (the pattern is similar for the first ~10 sentences I tried):

To debug, I looked at whether the same kind of thing comes up in the demo Colab notebook, and it seems like it is. Here's what I see if I see for the range of the embeddings for those two sentences:

# NEW: added this, with different messages
messages = ["cats are great!", "sometimes models are confusing"]
values, indices, dense_shape = process_to_IDs_in_sparse_format(sp, messages)

with tf.Session() as session:
 session.run([tf.global_variables_initializer(), tf.tables_initializer()])
 message_embeddings = session.run(
 encodings,
 feed_dict=input_placeholder.values: values,
 input_placeholder.indices: indices,
 input_placeholder.dense_shape: dense_shape)

 for i, message_embedding in enumerate(np.array(message_embeddings).tolist()):
 print("Message: ".format(messages[i]))
 print("Embedding size: ".format(len(message_embedding)))
 message_embedding_snippet = ", ".join(
 (str(x) for x in message_embedding[:3]))
 print("Embedding: [, ...]n".format(message_embedding_snippet))
 # NEW: added this, to show the range of the embedding output
 print("Embedding range: [, ]".format(min(message_embedding), max(message_embedding)))

And the output shows:

Message: cats are great!
Embedding range: [-0.05904272198677063, 0.05903803929686546]

Message: sometimes models are confusing
Embedding range: [-0.060731519013643265, 0.06075377017259598]

So this again isn't what I'm expecting - the range is more narrow than I'd expect. I thought this might be a TF convention that I missed, but couldn't see it in the TFHub page or the guide to text embeddings or in the paper so am not sure where else to look without digging into the training code.

The colab notebook example code has an example sentence that says:

Universal Sentence Encoder embeddings also support short paragraphs.
There is no hard limit on how long the paragraph is. Roughly, the
longer the more 'diluted' the embedding will be.

But the range of the embedding is roughly the same for all the other examples in the colab, even one word examples.

I'm assuming this range is not just arbitrary, and it does make sense to me that the range is centered in zero and small, but I'm trying to understand how this scale came to be.

The output of the universal sentence encoder is a vector of length 512, with an L2 norm of (approximately) 1.0. You can check this by calculating the inner product

ip = 0
for i in range(512):
 ip += message_embeddings[0][i] * message_embeddings[0][i]

print(ip)

> 1.0000000807544893

The implications are that:

• Most values are likely to be in a narrow range centered around zero


• The largest possible single value in the vector is 1.0 - and this would only happen if all other values are exactly 0.


• Similarly the smallest possible value is -1.


• If we take a random vector of length 512, with values distributed uniformly, and then normalize it to unit magnitude, we expect to see values in a range similar to what you see.

rand_uniform = np.random.uniform(-1, 1, 512)
l2 = np.linalg.norm(rand_uniform)
plt.plot(rand_uniform / l2, 'b.')
axes = plt.gca()
axes.set_ylim([-0.5, 0.5])

Judging visually, the distribution of excitations does not look uniform, but rather is biased toward extremes.