Styjun

What is the meaning of "prairie-dog" in this sentence?

What does the multimeter dial do internally?

Need a non-volatile memory IC with near unlimited read/write operations capability

How do I use my adapted PS2 keyboard & mouse on a windows 10 computer?

How should I ask for a "pint" in countries that use metric?

Gory anime with pink haired girl escaping an asylum

What was the nature of the known bugs in the Space Shuttle software?

Is this car delivery via Ebay Motors on Craigslist a scam?

When do flights get cancelled due to fog?

How to understand flavors and when to use combination of them?

Editing start vertex of multiple elements synchronously using QGIS?

My professor has told me he will be the corresponding author. Will it hurt my future career?

Is it ok for parents to kiss and romance with each other while their 2- to 8-year-old child watches?

Curly braces adjustment in tikz?

Why do people prefer metropolitan areas, considering monsters and villains?

Forward DNS request to my work's jump server

What are the effects of abstaining from eating a certain flavor?

Is this really the Saturn V computer only, or are there other systems here as well?

Strong Password Detection in Python

QR codes, do people use them?

How can I use my cell phone's light as a reading light?

Is space division multiplexing really multiplexing?

Why SQL does not use the indexed view?

Sense of humor in your sci-fi stories

How to be able to process a large JSON response?

Deserialising a JSON responseJSON Reciever EndpointJson response getting truncated when getting from SalesforceExtra Backslash in Json String when exposing webserviceIllegal Value for primitive when trying to parse the json dataBest way to parse JSON Response from Google Maps?Create Salesforce Records from JSON responseJSON Parsing of REST responseto remove null value from the JSON responseDynamic UI based on Large JSON response / Web UI based on JSON Schema implementation

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;

1

We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?

json rest governorlimits

share|improve this question

edited Mar 25 at 18:13

Sander de Jong

asked Mar 25 at 17:48

Sander de JongSander de Jong

1,43622 gold badges1515 silver badges3838 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?
  
  – Jayant Das
  Mar 25 at 18:05
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @JayantDas I have added clarification to my question.
  
  – Sander de Jong
  Mar 25 at 18:13
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.
  
  – Jayant Das
  Mar 25 at 18:14
  

  
  
  
  

add a comment | 

1

We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?

json rest governorlimits

share|improve this question

edited Mar 25 at 18:13

Sander de Jong

asked Mar 25 at 17:48

Sander de JongSander de Jong

1,43622 gold badges1515 silver badges3838 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Just to clarify, you are calling a Salesforce REST service from an external client or you are calling an external REST service from within Salesforce (apex)?
  
  – Jayant Das
  Mar 25 at 18:05
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @JayantDas I have added clarification to my question.
  
  – Sander de Jong
  Mar 25 at 18:13
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Thanks. Had thought that you are calling from Salesforce but good to get that clarified. Derek's answer is what I would think addresses this scenario.
  
  – Jayant Das
  Mar 25 at 18:14
  

  
  
  
  

add a comment | 

We're calling an external REST service from Salesforce and are receiving a JSON response of more than 45 MB. Has anyone attempted anything like this?

json rest governorlimits

share|improve this question

edited Mar 25 at 18:13

Sander de Jong

asked Mar 25 at 17:48

Sander de JongSander de Jong

1,43622 gold badges1515 silver badges3838 bronze badges

share|improve this question

edited Mar 25 at 18:13

Sander de Jong

asked Mar 25 at 17:48

Sander de JongSander de Jong

1,43622 gold badges1515 silver badges3838 bronze badges

share|improve this question

edited Mar 25 at 18:13

Sander de Jong

asked Mar 25 at 17:48

Sander de JongSander de Jong

1,43622 gold badges1515 silver badges3838 bronze badges

asked Mar 25 at 17:48

Sander de JongSander de Jong

1,43622 gold badges1515 silver badges3838 bronze badges

asked Mar 25 at 17:48

Sander de JongSander de Jong

1,43622 gold badges1515 silver badges3838 bronze badges

2 Answers
2

active

oldest

votes

4

There's just no way to cram that much data into Salesforce as-is.

I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).

As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.

Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs

share|improve this answer

answered Mar 25 at 17:58

Derek FDerek F

21.9k66 gold badges2525 silver badges5353 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.
  
  – Sander de Jong
  Mar 25 at 18:11
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.
  
  – Sander de Jong
  Mar 25 at 18:14
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.
  
  – Derek F
  Mar 25 at 21:06
  

  
  
  
  

add a comment | 

0

In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.

share|improve this answer

answered Mar 25 at 17:56

sfdcfoxsfdcfox

277k1414 gold badges225225 silver badges476476 bronze badges

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "459"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsalesforce.stackexchange.com%2fquestions%2f255209%2fhow-to-be-able-to-process-a-large-json-response%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

4

There's just no way to cram that much data into Salesforce as-is.

I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).

As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.

Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs

share|improve this answer

answered Mar 25 at 17:58

Derek FDerek F

21.9k66 gold badges2525 silver badges5353 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.
  
  – Sander de Jong
  Mar 25 at 18:11
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.
  
  – Sander de Jong
  Mar 25 at 18:14
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.
  
  – Derek F
  Mar 25 at 21:06
  

  
  
  
  

add a comment | 

4

There's just no way to cram that much data into Salesforce as-is.

I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).

As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.

Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs

share|improve this answer

answered Mar 25 at 17:58

Derek FDerek F

21.9k66 gold badges2525 silver badges5353 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I like your suggestion of having the middle layer call Salesforce's APIs. I don't know if the middle layer will be able to handle that kind of load. An alternative would be to have the middle layer implement a paging mechanism, but this would also mean more work on the Salesforce side.
  
  – Sander de Jong
  Mar 25 at 18:11
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I also think that doing tens of thousands of REST calls to Salesforce will not be as efficient as processing 45 times 1 MB JSON files, for example.
  
  – Sander de Jong
  Mar 25 at 18:14
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  @SanderdeJong Depending on what precisely you're doing, you could create multiple records in Salesforce with a single request, or use the composite resource to package up multiple API calls into a single call.
  
  – Derek F
  Mar 25 at 21:06
  

  
  
  
  

add a comment | 

There's just no way to cram that much data into Salesforce as-is.

I'd imagine you'd need to introduce a layer in the middle to break that JSON up into more manageable pieces, and also make use of async processing (probably using Queueable).

As I'm sure you know, the transaction limit on heap size is 6MB (12MB Async), and that space needs to hold the incoming response, store it again (plus overhead) when you deserialize it, and have space left over to do whatever processing you need to do.

Another option would be to have the middle layer (Heroku, some VM on some cloud provider, etc...) do the processing for you, and then create/update/delete whatever it is that you need to do via accessing Salesforce's APIs

share|improve this answer

answered Mar 25 at 17:58

Derek FDerek F

21.9k66 gold badges2525 silver badges5353 bronze badges

share|improve this answer

answered Mar 25 at 17:58

Derek FDerek F

21.9k66 gold badges2525 silver badges5353 bronze badges

answered Mar 25 at 17:58

Derek FDerek F

21.9k66 gold badges2525 silver badges5353 bronze badges

answered Mar 25 at 17:58

Derek FDerek F

21.9k66 gold badges2525 silver badges5353 bronze badges

0

In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.

share|improve this answer

answered Mar 25 at 17:56

sfdcfoxsfdcfox

277k1414 gold badges225225 silver badges476476 bronze badges

add a comment | 

0

In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.

share|improve this answer

answered Mar 25 at 17:56

sfdcfoxsfdcfox

277k1414 gold badges225225 silver badges476476 bronze badges

add a comment | 

In Apex, no chance. The maximum response size is limited to 6/12/36MB (depending on context). You'd have to do client-side processing (e.g. Visualforce via the AJAX Proxy) in order to handle this amount of data.

share|improve this answer

answered Mar 25 at 17:56

sfdcfoxsfdcfox

277k1414 gold badges225225 silver badges476476 bronze badges

share|improve this answer

answered Mar 25 at 17:56

sfdcfoxsfdcfox

277k1414 gold badges225225 silver badges476476 bronze badges

answered Mar 25 at 17:56

sfdcfoxsfdcfox

277k1414 gold badges225225 silver badges476476 bronze badges

answered Mar 25 at 17:56

sfdcfoxsfdcfox

277k1414 gold badges225225 silver badges476476 bronze badges