When using Beam IO ReadFromPubSub module, can you pull messages with attributes in Python? It's unclear if its supportedHow can you profile a Python script?How can I get a list of locally installed Python modules?Dataflow reading using PubSubIO is really slowHow often does PubsubIO.readStrings pull from subscriptionDatastoreException: A non-transactional commit may not contain multiple mutations affecting the same entityDataflow autoscale does not boost performanceHow does PubSubIO determine reading speed when using DataflowRunnerPubsubIO does not output custom timestamp attribute as context.timestamp when running with DataflowRunner and Dataflow serviceReplaying data into Apache Beam pipeline over Google Cloud Pub/Sub without overloading other subscribersBeam / DataFlow ::ReadFromPubSub(id_label) :: Unexpected behavior
Where can I find visible/radio telescopic observations of the center of the Milky Way galaxy?
Why did David Cameron offer a referendum on the European Union?
In general, would I need to season a meat when making a sauce?
Why are C64 games inconsistent with which joystick port they use?
What is a Centaur Thief's climbing speed?
Where have Brexit voters gone?
Need to understand my home electrical meter to see why bill is so high and/or if neighbor is on same meter
Should breaking down something like a door be adjudicated as an attempt to beat its AC and HP, or as an ability check against a set DC?
Why would Ryanair allow me to book this journey through a third party, but not through their own website?
Compaq Portable vs IBM 5155 Portable PC
How to illustrate the Mean Value theorem?
Count Even Digits In Number
Should I disclose a colleague's illness (that I should not know) when others badmouth him
Why were helmets and other body armour not commonplace in the 1800s?
A Riley Respite
Using credit/debit card details vs swiping a card in a payment (credit card) terminal
How to respond to an upset student?
The art of clickbait captions
Is the taxi route omitted in low visibility (LVP)?
Have 1.5% of all nuclear reactors ever built melted down?
Can a person survive on blood in place of water?
Why does Mjolnir fall down in Age of Ultron but not in Endgame?
Did 20% of US soldiers in Vietnam use heroin, 95% of whom quit afterwards?
Which melee weapons have the Two-Handed property, but lack Heavy and Special?
When using Beam IO ReadFromPubSub module, can you pull messages with attributes in Python? It's unclear if its supported
How can you profile a Python script?How can I get a list of locally installed Python modules?Dataflow reading using PubSubIO is really slowHow often does PubsubIO.readStrings pull from subscriptionDatastoreException: A non-transactional commit may not contain multiple mutations affecting the same entityDataflow autoscale does not boost performanceHow does PubSubIO determine reading speed when using DataflowRunnerPubsubIO does not output custom timestamp attribute as context.timestamp when running with DataflowRunner and Dataflow serviceReplaying data into Apache Beam pipeline over Google Cloud Pub/Sub without overloading other subscribersBeam / DataFlow ::ReadFromPubSub(id_label) :: Unexpected behavior
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
Trying to pull messages with attributes stored in PubSub into a Beam pipeline. I'm wondering if support has been added for Python and that's why I'm unable to read them. I see that it exists in Java.
pipeline_options = PipelineOptions()
pipeline_options.view_as(StandardOptions).streaming = True
pipeline = beam.Pipeline(options=pipeline_options)
messages = (pipeline | beam.io.ReadFromPubSub(subscription=subscription_name).with_output_types(bytes))
def printattr(element):
print(element.attributes)
lines = messages | 'printattr' >> beam.Map(printattr)
result = pipeline.run()
result.wait_until_finish()
Expected to be able to list the attributes- data:
b'Message number 1109'
- attributes: {
- "_comments": "nan",
- "_direction": "SE",
- "_fromst": "Harlem",
- "_last_updt": "2019-03-20 21:11:02.0",
- "_length": "0.56",
- "_lif_lat": "41.9809967484",
- "_lit_lat": "41.9787314076",
- "_lit_lon": "-87.7964600566",
- "_strheading": "W",
- "_tost": "Oak Park",
, but I can only seem to access information stored in the data field- not the attributes.
python google-cloud-pubsub apache-beam-io
edited Mar 24 at 5:20
Francis
7721325
asked Mar 24 at 4:17
cloudpythoncloudpython
233
add a comment |
Trying to pull messages with attributes stored in PubSub into a Beam pipeline. I'm wondering if support has been added for Python and that's why I'm unable to read them. I see that it exists in Java.
pipeline_options = PipelineOptions()
pipeline_options.view_as(StandardOptions).streaming = True
pipeline = beam.Pipeline(options=pipeline_options)
messages = (pipeline | beam.io.ReadFromPubSub(subscription=subscription_name).with_output_types(bytes))
def printattr(element):
print(element.attributes)
lines = messages | 'printattr' >> beam.Map(printattr)
result = pipeline.run()
result.wait_until_finish()
Expected to be able to list the attributes- data:
b'Message number 1109'
- attributes: {
- "_comments": "nan",
- "_direction": "SE",
- "_fromst": "Harlem",
- "_last_updt": "2019-03-20 21:11:02.0",
- "_length": "0.56",
- "_lif_lat": "41.9809967484",
- "_lit_lat": "41.9787314076",
- "_lit_lon": "-87.7964600566",
- "_strheading": "W",
- "_tost": "Oak Park",
, but I can only seem to access information stored in the data field- not the attributes.
python google-cloud-pubsub apache-beam-io
edited Mar 24 at 5:20
Francis
7721325
asked Mar 24 at 4:17
cloudpythoncloudpython
233
add a comment |
Trying to pull messages with attributes stored in PubSub into a Beam pipeline. I'm wondering if support has been added for Python and that's why I'm unable to read them. I see that it exists in Java.
pipeline_options = PipelineOptions()
pipeline_options.view_as(StandardOptions).streaming = True
pipeline = beam.Pipeline(options=pipeline_options)
messages = (pipeline | beam.io.ReadFromPubSub(subscription=subscription_name).with_output_types(bytes))
def printattr(element):
print(element.attributes)
lines = messages | 'printattr' >> beam.Map(printattr)
result = pipeline.run()
result.wait_until_finish()
Expected to be able to list the attributes- data:
b'Message number 1109'
- attributes: {
- "_comments": "nan",
- "_direction": "SE",
- "_fromst": "Harlem",
- "_last_updt": "2019-03-20 21:11:02.0",
- "_length": "0.56",
- "_lif_lat": "41.9809967484",
- "_lit_lat": "41.9787314076",
- "_lit_lon": "-87.7964600566",
- "_strheading": "W",
- "_tost": "Oak Park",
, but I can only seem to access information stored in the data field- not the attributes.
python google-cloud-pubsub apache-beam-io
edited Mar 24 at 5:20
Francis
7721325
asked Mar 24 at 4:17
cloudpythoncloudpython
233
Trying to pull messages with attributes stored in PubSub into a Beam pipeline. I'm wondering if support has been added for Python and that's why I'm unable to read them. I see that it exists in Java.
pipeline_options = PipelineOptions()
pipeline_options.view_as(StandardOptions).streaming = True
pipeline = beam.Pipeline(options=pipeline_options)
messages = (pipeline | beam.io.ReadFromPubSub(subscription=subscription_name).with_output_types(bytes))
def printattr(element):
print(element.attributes)
lines = messages | 'printattr' >> beam.Map(printattr)
result = pipeline.run()
result.wait_until_finish()
Expected to be able to list the attributes- data:
b'Message number 1109'
- attributes: {
- "_comments": "nan",
- "_direction": "SE",
- "_fromst": "Harlem",
- "_last_updt": "2019-03-20 21:11:02.0",
- "_length": "0.56",
- "_lif_lat": "41.9809967484",
- "_lit_lat": "41.9787314076",
- "_lit_lon": "-87.7964600566",
- "_strheading": "W",
- "_tost": "Oak Park",
, but I can only seem to access information stored in the data field- not the attributes.
python google-cloud-pubsub apache-beam-io
python google-cloud-pubsub apache-beam-io
edited Mar 24 at 5:20
Francis
7721325
asked Mar 24 at 4:17
cloudpythoncloudpython
233
edited Mar 24 at 5:20
Francis
7721325
asked Mar 24 at 4:17
cloudpythoncloudpython
233
edited Mar 24 at 5:20
Francis
7721325
edited Mar 24 at 5:20
Francis
7721325
edited Mar 24 at 5:20
Francis
7721325
7721325
asked Mar 24 at 4:17
cloudpythoncloudpython
233
asked Mar 24 at 4:17
cloudpythoncloudpython
233
asked Mar 24 at 4:17
cloudpythoncloudpython
233
233
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
After reviewing the documentation: https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.io.gcp.pubsub.html, I was able to see an additional argument to pass in to ReadFromPubSub.
The 'with_attributes=True' needed to be set, otherwise, you just get the data fields.
Hopefully this helps someone else who may get stuck or just be tired :)
answered Mar 24 at 4:29
cloudpythoncloudpython
233
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55320682%2fwhen-using-beam-io-readfrompubsub-module-can-you-pull-messages-with-attributes%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
After reviewing the documentation: https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.io.gcp.pubsub.html, I was able to see an additional argument to pass in to ReadFromPubSub.
The 'with_attributes=True' needed to be set, otherwise, you just get the data fields.
Hopefully this helps someone else who may get stuck or just be tired :)
answered Mar 24 at 4:29
cloudpythoncloudpython
233
add a comment |
After reviewing the documentation: https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.io.gcp.pubsub.html, I was able to see an additional argument to pass in to ReadFromPubSub.
The 'with_attributes=True' needed to be set, otherwise, you just get the data fields.
Hopefully this helps someone else who may get stuck or just be tired :)
answered Mar 24 at 4:29
cloudpythoncloudpython
233
add a comment |
After reviewing the documentation: https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.io.gcp.pubsub.html, I was able to see an additional argument to pass in to ReadFromPubSub.
The 'with_attributes=True' needed to be set, otherwise, you just get the data fields.
Hopefully this helps someone else who may get stuck or just be tired :)
answered Mar 24 at 4:29
cloudpythoncloudpython
233
After reviewing the documentation: https://beam.apache.org/releases/pydoc/2.11.0/apache_beam.io.gcp.pubsub.html, I was able to see an additional argument to pass in to ReadFromPubSub.
The 'with_attributes=True' needed to be set, otherwise, you just get the data fields.
Hopefully this helps someone else who may get stuck or just be tired :)
answered Mar 24 at 4:29
cloudpythoncloudpython
233
answered Mar 24 at 4:29
cloudpythoncloudpython
233
answered Mar 24 at 4:29
cloudpythoncloudpython
233
answered Mar 24 at 4:29
cloudpythoncloudpython
233
233
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55320682%2fwhen-using-beam-io-readfrompubsub-module-can-you-pull-messages-with-attributes%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
