How to implement a multi-tenant database for spring cloud data flowMulti-tenant job repository in spring-batchSpring Cloud Data Flow and SCDF Kafka DataSourceHow to properly create Spring Cloud Task with custom parameters?Implementing multi tenancy on Spring Data JPAJava Spring data in MultiTenant architecture/PaaS SAP Hana CloudSpring cloud data flow with spring batch job - scaling considerationsLaunching Spring Batch Task from Spring Cloud Data FlowSpring Data Redis with multi tenancySCDF Local data flow server
Reducing contention in thread-safe LruCache
How to render "have ideas above his station" into German
Would getting a natural 20 with a penalty still count as a critical hit?
Subgroup generated by a subgroup and a conjugate of it
What exactly happened to the 18 crew members who were reported as "missing" in "Q Who"?
Will some rockets really collapse under their own weight?
How do I answer an interview question about how to handle a hard deadline I won't be able to meet?
"I like watching" vs "I like to watch" What's the difference?
What are some tips and tricks for finding the cheapest flight when luggage and other fees are not revealed until far into the booking process?
Why is the battery jumpered to a resistor in this schematic?
What's the relationship betweeen MS-DOS and XENIX?
Adjective or adverb before another adjective
Is the Microsoft recommendation to use C# properties applicable to game development?
How to use the passive form to say "This flower was watered."
The Lucky House
Alignement of different align environment
Problem with GFCI at start of circuit with both lights and two receptacles
Parse a simple key=value config file in C
Unsolved Problems due to Lack of Computational Power
How should you gracefully leave a company you helped start?
Expressing a chain of boolean ORs using ILP
Yes/ No : The sum of two ideals of a ring R is an ideal of R
Why does this image of cyclocarbon look like a nonagon?
A+ rating still unsecure by Google Chrome's opinion
How to implement a multi-tenant database for spring cloud data flow
Multi-tenant job repository in spring-batchSpring Cloud Data Flow and SCDF Kafka DataSourceHow to properly create Spring Cloud Task with custom parameters?Implementing multi tenancy on Spring Data JPAJava Spring data in MultiTenant architecture/PaaS SAP Hana CloudSpring cloud data flow with spring batch job - scaling considerationsLaunching Spring Batch Task from Spring Cloud Data FlowSpring Data Redis with multi tenancySCDF Local data flow server
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
We would like to implement a multi-tenant solution for SCDF for which each tenant may have unique task definitions / etc. Ideally we only want a single SCDF server (as opposed to setting up an SCDF server for each tenant), as pictured:
Is this possible or is the only way to achieve isolation of the data between tenants to have separate data flow server instances?
spring spring-cloud-dataflow spring-cloud-task
asked Mar 27 at 12:40
GaZGaZ
1,76220 silver badges37 bronze badges
add a comment |
We would like to implement a multi-tenant solution for SCDF for which each tenant may have unique task definitions / etc. Ideally we only want a single SCDF server (as opposed to setting up an SCDF server for each tenant), as pictured:
Is this possible or is the only way to achieve isolation of the data between tenants to have separate data flow server instances?
spring spring-cloud-dataflow spring-cloud-task
asked Mar 27 at 12:40
GaZGaZ
1,76220 silver badges37 bronze badges
add a comment |
We would like to implement a multi-tenant solution for SCDF for which each tenant may have unique task definitions / etc. Ideally we only want a single SCDF server (as opposed to setting up an SCDF server for each tenant), as pictured:
Is this possible or is the only way to achieve isolation of the data between tenants to have separate data flow server instances?
spring spring-cloud-dataflow spring-cloud-task
asked Mar 27 at 12:40
GaZGaZ
1,76220 silver badges37 bronze badges
We would like to implement a multi-tenant solution for SCDF for which each tenant may have unique task definitions / etc. Ideally we only want a single SCDF server (as opposed to setting up an SCDF server for each tenant), as pictured:
Is this possible or is the only way to achieve isolation of the data between tenants to have separate data flow server instances?
spring spring-cloud-dataflow spring-cloud-task
spring spring-cloud-dataflow spring-cloud-task
asked Mar 27 at 12:40
GaZGaZ
1,76220 silver badges37 bronze badges
asked Mar 27 at 12:40
GaZGaZ
1,76220 silver badges37 bronze badges
asked Mar 27 at 12:40
GaZGaZ
1,76220 silver badges37 bronze badges
asked Mar 27 at 12:40
GaZGaZ
1,76220 silver badges37 bronze badges
asked Mar 27 at 12:40
GaZGaZ
1,76220 silver badges37 bronze badges
1,76220 silver badges37 bronze badges
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
What you're attempting here is not possible today. You'd have to provision SCDF for each tenant. In cloud platforms like Kubernetes or Cloud Foundry, it is recommended because you can access-control the tenants through "namespace" and "org/space" isolation respectively. On this foundation, the platforms provide a more robust separation through RBAC assignments for each user in the Tenant.
A little bit of more background as to why we do this today. SCDF and the Task/Job repositories are coupled in the sense that the Dashboard and the other client tools interact with the same datasource to provide the consistent UX to monitor and manage the data pipelines centrally. With the recent multi-platform backends support for Tasks, you're still expected to use a common datasource in the current design.
All that said, we are looking into improving to allow users to have a database with schemas prefixed with an identifier [see: spring-cloud/spring-cloud-dataflow#2048]. With that in place, it would be possible to then filter by the identifier-specific task/job executions and likewise track them as isolated units of operations within the single SCDF instance.
However, it may not scale for cloud deployments. Each of the tenant isolation boundaries, for instance, a "namespace" in Kubernetes needs to have enough resources (cpu/memory/disk) to handle "multiple" tenant deployments of task/batch apps. If you don't autoscale the resource capacity, you'd have deployment failures.
Maybe you could help with describing your requirements in some more detail, so we could relate to why this could still be useful. Please also share how you're going to design the resource allocations in the underlying deployment platform - feel free to comment in #2048.
answered Mar 27 at 17:05
Sabby AnandanSabby Anandan
3,6851 gold badge5 silver badges14 bronze badges
Thanks for the info, Sabby. We will reconsider the architecture and consider the pros and cons of a single repository vs multiple SCDFs (one per tenant). I don't think that a custom table prefix feature mentioned in 2048 would be helpful for our case. The primary requirement / functionality we're looking for is the ability to launch Tasks as pods. We will have a custom GUI for launching the tasks, so I guess it should be possible for SCDF to have a single, separate datasource from the tenants in the diagram above.
– GaZ
Mar 27 at 20:19
Thanks for the clarification. The tenant in your case can be mapped to a platform-account in SCDF, which is nothing but a tenant differentiated by something unique. Let's say a tenant is distinguished by "namespace" in Kubernetes, and that can be a set of configuration properties see: [2852#issuecomment-462957754]. With multiple platform configuration in SCDF, you can select and launch the Task against them - this is supported today.
– Sabby Anandan
Mar 27 at 21:21
However, it doesn't provide the mechanism to connect each platform against a different database. A common datasource is a requirement today. If we add support for it, SCDF should then be able to query based on the platform identifier to derive the corresponding datasource to launch and retrieve Task executions.
– Sabby Anandan
Mar 27 at 21:23
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55377434%2fhow-to-implement-a-multi-tenant-database-for-spring-cloud-data-flow%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
What you're attempting here is not possible today. You'd have to provision SCDF for each tenant. In cloud platforms like Kubernetes or Cloud Foundry, it is recommended because you can access-control the tenants through "namespace" and "org/space" isolation respectively. On this foundation, the platforms provide a more robust separation through RBAC assignments for each user in the Tenant.
A little bit of more background as to why we do this today. SCDF and the Task/Job repositories are coupled in the sense that the Dashboard and the other client tools interact with the same datasource to provide the consistent UX to monitor and manage the data pipelines centrally. With the recent multi-platform backends support for Tasks, you're still expected to use a common datasource in the current design.
All that said, we are looking into improving to allow users to have a database with schemas prefixed with an identifier [see: spring-cloud/spring-cloud-dataflow#2048]. With that in place, it would be possible to then filter by the identifier-specific task/job executions and likewise track them as isolated units of operations within the single SCDF instance.
However, it may not scale for cloud deployments. Each of the tenant isolation boundaries, for instance, a "namespace" in Kubernetes needs to have enough resources (cpu/memory/disk) to handle "multiple" tenant deployments of task/batch apps. If you don't autoscale the resource capacity, you'd have deployment failures.
Maybe you could help with describing your requirements in some more detail, so we could relate to why this could still be useful. Please also share how you're going to design the resource allocations in the underlying deployment platform - feel free to comment in #2048.
answered Mar 27 at 17:05
Sabby AnandanSabby Anandan
3,6851 gold badge5 silver badges14 bronze badges
Thanks for the info, Sabby. We will reconsider the architecture and consider the pros and cons of a single repository vs multiple SCDFs (one per tenant). I don't think that a custom table prefix feature mentioned in 2048 would be helpful for our case. The primary requirement / functionality we're looking for is the ability to launch Tasks as pods. We will have a custom GUI for launching the tasks, so I guess it should be possible for SCDF to have a single, separate datasource from the tenants in the diagram above.
– GaZ
Mar 27 at 20:19
Thanks for the clarification. The tenant in your case can be mapped to a platform-account in SCDF, which is nothing but a tenant differentiated by something unique. Let's say a tenant is distinguished by "namespace" in Kubernetes, and that can be a set of configuration properties see: [2852#issuecomment-462957754]. With multiple platform configuration in SCDF, you can select and launch the Task against them - this is supported today.
– Sabby Anandan
Mar 27 at 21:21
However, it doesn't provide the mechanism to connect each platform against a different database. A common datasource is a requirement today. If we add support for it, SCDF should then be able to query based on the platform identifier to derive the corresponding datasource to launch and retrieve Task executions.
– Sabby Anandan
Mar 27 at 21:23
add a comment |
What you're attempting here is not possible today. You'd have to provision SCDF for each tenant. In cloud platforms like Kubernetes or Cloud Foundry, it is recommended because you can access-control the tenants through "namespace" and "org/space" isolation respectively. On this foundation, the platforms provide a more robust separation through RBAC assignments for each user in the Tenant.
A little bit of more background as to why we do this today. SCDF and the Task/Job repositories are coupled in the sense that the Dashboard and the other client tools interact with the same datasource to provide the consistent UX to monitor and manage the data pipelines centrally. With the recent multi-platform backends support for Tasks, you're still expected to use a common datasource in the current design.
All that said, we are looking into improving to allow users to have a database with schemas prefixed with an identifier [see: spring-cloud/spring-cloud-dataflow#2048]. With that in place, it would be possible to then filter by the identifier-specific task/job executions and likewise track them as isolated units of operations within the single SCDF instance.
However, it may not scale for cloud deployments. Each of the tenant isolation boundaries, for instance, a "namespace" in Kubernetes needs to have enough resources (cpu/memory/disk) to handle "multiple" tenant deployments of task/batch apps. If you don't autoscale the resource capacity, you'd have deployment failures.
Maybe you could help with describing your requirements in some more detail, so we could relate to why this could still be useful. Please also share how you're going to design the resource allocations in the underlying deployment platform - feel free to comment in #2048.
answered Mar 27 at 17:05
Sabby AnandanSabby Anandan
3,6851 gold badge5 silver badges14 bronze badges
Thanks for the info, Sabby. We will reconsider the architecture and consider the pros and cons of a single repository vs multiple SCDFs (one per tenant). I don't think that a custom table prefix feature mentioned in 2048 would be helpful for our case. The primary requirement / functionality we're looking for is the ability to launch Tasks as pods. We will have a custom GUI for launching the tasks, so I guess it should be possible for SCDF to have a single, separate datasource from the tenants in the diagram above.
– GaZ
Mar 27 at 20:19
Thanks for the clarification. The tenant in your case can be mapped to a platform-account in SCDF, which is nothing but a tenant differentiated by something unique. Let's say a tenant is distinguished by "namespace" in Kubernetes, and that can be a set of configuration properties see: [2852#issuecomment-462957754]. With multiple platform configuration in SCDF, you can select and launch the Task against them - this is supported today.
– Sabby Anandan
Mar 27 at 21:21
However, it doesn't provide the mechanism to connect each platform against a different database. A common datasource is a requirement today. If we add support for it, SCDF should then be able to query based on the platform identifier to derive the corresponding datasource to launch and retrieve Task executions.
– Sabby Anandan
Mar 27 at 21:23
add a comment |
What you're attempting here is not possible today. You'd have to provision SCDF for each tenant. In cloud platforms like Kubernetes or Cloud Foundry, it is recommended because you can access-control the tenants through "namespace" and "org/space" isolation respectively. On this foundation, the platforms provide a more robust separation through RBAC assignments for each user in the Tenant.
A little bit of more background as to why we do this today. SCDF and the Task/Job repositories are coupled in the sense that the Dashboard and the other client tools interact with the same datasource to provide the consistent UX to monitor and manage the data pipelines centrally. With the recent multi-platform backends support for Tasks, you're still expected to use a common datasource in the current design.
All that said, we are looking into improving to allow users to have a database with schemas prefixed with an identifier [see: spring-cloud/spring-cloud-dataflow#2048]. With that in place, it would be possible to then filter by the identifier-specific task/job executions and likewise track them as isolated units of operations within the single SCDF instance.
However, it may not scale for cloud deployments. Each of the tenant isolation boundaries, for instance, a "namespace" in Kubernetes needs to have enough resources (cpu/memory/disk) to handle "multiple" tenant deployments of task/batch apps. If you don't autoscale the resource capacity, you'd have deployment failures.
Maybe you could help with describing your requirements in some more detail, so we could relate to why this could still be useful. Please also share how you're going to design the resource allocations in the underlying deployment platform - feel free to comment in #2048.
answered Mar 27 at 17:05
Sabby AnandanSabby Anandan
3,6851 gold badge5 silver badges14 bronze badges
What you're attempting here is not possible today. You'd have to provision SCDF for each tenant. In cloud platforms like Kubernetes or Cloud Foundry, it is recommended because you can access-control the tenants through "namespace" and "org/space" isolation respectively. On this foundation, the platforms provide a more robust separation through RBAC assignments for each user in the Tenant.
A little bit of more background as to why we do this today. SCDF and the Task/Job repositories are coupled in the sense that the Dashboard and the other client tools interact with the same datasource to provide the consistent UX to monitor and manage the data pipelines centrally. With the recent multi-platform backends support for Tasks, you're still expected to use a common datasource in the current design.
All that said, we are looking into improving to allow users to have a database with schemas prefixed with an identifier [see: spring-cloud/spring-cloud-dataflow#2048]. With that in place, it would be possible to then filter by the identifier-specific task/job executions and likewise track them as isolated units of operations within the single SCDF instance.
However, it may not scale for cloud deployments. Each of the tenant isolation boundaries, for instance, a "namespace" in Kubernetes needs to have enough resources (cpu/memory/disk) to handle "multiple" tenant deployments of task/batch apps. If you don't autoscale the resource capacity, you'd have deployment failures.
Maybe you could help with describing your requirements in some more detail, so we could relate to why this could still be useful. Please also share how you're going to design the resource allocations in the underlying deployment platform - feel free to comment in #2048.
answered Mar 27 at 17:05
Sabby AnandanSabby Anandan
3,6851 gold badge5 silver badges14 bronze badges
answered Mar 27 at 17:05
Sabby AnandanSabby Anandan
3,6851 gold badge5 silver badges14 bronze badges
answered Mar 27 at 17:05
Sabby AnandanSabby Anandan
3,6851 gold badge5 silver badges14 bronze badges
answered Mar 27 at 17:05
Sabby AnandanSabby Anandan
3,6851 gold badge5 silver badges14 bronze badges
3,6851 gold badge5 silver badges14 bronze badges
Thanks for the info, Sabby. We will reconsider the architecture and consider the pros and cons of a single repository vs multiple SCDFs (one per tenant). I don't think that a custom table prefix feature mentioned in 2048 would be helpful for our case. The primary requirement / functionality we're looking for is the ability to launch Tasks as pods. We will have a custom GUI for launching the tasks, so I guess it should be possible for SCDF to have a single, separate datasource from the tenants in the diagram above.
– GaZ
Mar 27 at 20:19
Thanks for the clarification. The tenant in your case can be mapped to a platform-account in SCDF, which is nothing but a tenant differentiated by something unique. Let's say a tenant is distinguished by "namespace" in Kubernetes, and that can be a set of configuration properties see: [2852#issuecomment-462957754]. With multiple platform configuration in SCDF, you can select and launch the Task against them - this is supported today.
– Sabby Anandan
Mar 27 at 21:21
However, it doesn't provide the mechanism to connect each platform against a different database. A common datasource is a requirement today. If we add support for it, SCDF should then be able to query based on the platform identifier to derive the corresponding datasource to launch and retrieve Task executions.
– Sabby Anandan
Mar 27 at 21:23
add a comment |
Thanks for the info, Sabby. We will reconsider the architecture and consider the pros and cons of a single repository vs multiple SCDFs (one per tenant). I don't think that a custom table prefix feature mentioned in 2048 would be helpful for our case. The primary requirement / functionality we're looking for is the ability to launch Tasks as pods. We will have a custom GUI for launching the tasks, so I guess it should be possible for SCDF to have a single, separate datasource from the tenants in the diagram above.
– GaZ
Mar 27 at 20:19
Thanks for the clarification. The tenant in your case can be mapped to a platform-account in SCDF, which is nothing but a tenant differentiated by something unique. Let's say a tenant is distinguished by "namespace" in Kubernetes, and that can be a set of configuration properties see: [2852#issuecomment-462957754]. With multiple platform configuration in SCDF, you can select and launch the Task against them - this is supported today.
– Sabby Anandan
Mar 27 at 21:21
However, it doesn't provide the mechanism to connect each platform against a different database. A common datasource is a requirement today. If we add support for it, SCDF should then be able to query based on the platform identifier to derive the corresponding datasource to launch and retrieve Task executions.
– Sabby Anandan
Mar 27 at 21:23
Thanks for the info, Sabby. We will reconsider the architecture and consider the pros and cons of a single repository vs multiple SCDFs (one per tenant). I don't think that a custom table prefix feature mentioned in 2048 would be helpful for our case. The primary requirement / functionality we're looking for is the ability to launch Tasks as pods. We will have a custom GUI for launching the tasks, so I guess it should be possible for SCDF to have a single, separate datasource from the tenants in the diagram above.
– GaZ
Mar 27 at 20:19
Thanks for the info, Sabby. We will reconsider the architecture and consider the pros and cons of a single repository vs multiple SCDFs (one per tenant). I don't think that a custom table prefix feature mentioned in 2048 would be helpful for our case. The primary requirement / functionality we're looking for is the ability to launch Tasks as pods. We will have a custom GUI for launching the tasks, so I guess it should be possible for SCDF to have a single, separate datasource from the tenants in the diagram above.
– GaZ
Mar 27 at 20:19
Thanks for the clarification. The tenant in your case can be mapped to a platform-account in SCDF, which is nothing but a tenant differentiated by something unique. Let's say a tenant is distinguished by "namespace" in Kubernetes, and that can be a set of configuration properties see: [2852#issuecomment-462957754]. With multiple platform configuration in SCDF, you can select and launch the Task against them - this is supported today.
– Sabby Anandan
Mar 27 at 21:21
Thanks for the clarification. The tenant in your case can be mapped to a platform-account in SCDF, which is nothing but a tenant differentiated by something unique. Let's say a tenant is distinguished by "namespace" in Kubernetes, and that can be a set of configuration properties see: [2852#issuecomment-462957754]. With multiple platform configuration in SCDF, you can select and launch the Task against them - this is supported today.
– Sabby Anandan
Mar 27 at 21:21
However, it doesn't provide the mechanism to connect each platform against a different database. A common datasource is a requirement today. If we add support for it, SCDF should then be able to query based on the platform identifier to derive the corresponding datasource to launch and retrieve Task executions.
– Sabby Anandan
Mar 27 at 21:23
However, it doesn't provide the mechanism to connect each platform against a different database. A common datasource is a requirement today. If we add support for it, SCDF should then be able to query based on the platform identifier to derive the corresponding datasource to launch and retrieve Task executions.
– Sabby Anandan
Mar 27 at 21:23
add a comment |
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Got a question that you can’t ask on public Stack Overflow? Learn more about sharing private information with Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55377434%2fhow-to-implement-a-multi-tenant-database-for-spring-cloud-data-flow%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
