Can you remove the UUID in partitioned parquet-files?How can I remove a trailing newline?How do you remove duplicates from a list whilst preserving order?How can you profile a Python script?When are you truly forced to use UUID as part of the design?How can I parse a YAML file in PythonHow can I open multiple files using “with open” in Python?How do you append to a file in Python?Partitioning the parquet fileImportError: No module named pymysql in python 2.7Partitioning strategy in Parquet and Spark
What is a "soap"?
Is there a fallacy about "appeal to 'big words'"?
What's the point of writing that I know will never be used or read?
Escape Velocity - Won't the orbital path just become larger with higher initial velocity?
What allows us to use imaginary numbers?
Why does Japan use the same type of AC power outlet as the US?
Why did IBM make the PC BIOS source code public?
Is there any official ruling on how characters go from 0th to 1st level in a class?
RAII wrapper for SQLite transactions
Adding things to bunches of things vs multiplication
What modifiers are added to the attack and damage rolls of this unique longbow from Waterdeep: Dragon Heist?
How do I answer an interview question about not meeting deadlines?
Can anybody tell me who this Pokemon is?
How do I ask for 2-3 days per week remote work in a job interview?
What is the fastest way to level past 95 in Diablo II?
Duplicate and slide edge (rip from boundary)
What should I do with the stock I own if I anticipate there will be a recession?
Mind ya, it's Homophones Everywhere!
Is there a word for returning to unpreparedness?
How to prevent criminal gangs from making/buying guns?
When did Bilbo and Frodo learn that Gandalf was a Maia?
Scam? Phone call from "Department of Social Security" asking me to call back
Quick destruction of a helium filled airship?
A+ rating still unsecure by Google Chrome's opinion
Can you remove the UUID in partitioned parquet-files?
How can I remove a trailing newline?How do you remove duplicates from a list whilst preserving order?How can you profile a Python script?When are you truly forced to use UUID as part of the design?How can I parse a YAML file in PythonHow can I open multiple files using “with open” in Python?How do you append to a file in Python?Partitioning the parquet fileImportError: No module named pymysql in python 2.7Partitioning strategy in Parquet and Spark
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
We store some data in partitioned parquet-files for a client. While that works without problems, the client ran into problems through the 255 character-limit in the filepath from windows. The folderstructure that is used for the partitioned data allows to clearly identify the stored data.
Changing the clients folderstructure isn`t really an option.
The idea we have is to shorten the filename by removing the UUID, in the best case while generating the file, if that isn`t possible by renaming it afterwards.
The enviorment can be summarized as following:
- Data Bricks Cluster with Python Notebooks
- Files are stored in an Azure Data Lake
Enivormentdata for the used Python:
- Compiler: GCC 5.4.0 20160609
- Build: ('default', 'Nov 23 2017 16:37:01')
- Implementation: CPython Python Version: 3.5.2
python uuid parquet cpython
asked Mar 27 at 12:23
BossBologneseBossBolognese
84 bronze badges
add a comment |
We store some data in partitioned parquet-files for a client. While that works without problems, the client ran into problems through the 255 character-limit in the filepath from windows. The folderstructure that is used for the partitioned data allows to clearly identify the stored data.
Changing the clients folderstructure isn`t really an option.
The idea we have is to shorten the filename by removing the UUID, in the best case while generating the file, if that isn`t possible by renaming it afterwards.
The enviorment can be summarized as following:
- Data Bricks Cluster with Python Notebooks
- Files are stored in an Azure Data Lake
Enivormentdata for the used Python:
- Compiler: GCC 5.4.0 20160609
- Build: ('default', 'Nov 23 2017 16:37:01')
- Implementation: CPython Python Version: 3.5.2
python uuid parquet cpython
asked Mar 27 at 12:23
BossBologneseBossBolognese
84 bronze badges
add a comment |
We store some data in partitioned parquet-files for a client. While that works without problems, the client ran into problems through the 255 character-limit in the filepath from windows. The folderstructure that is used for the partitioned data allows to clearly identify the stored data.
Changing the clients folderstructure isn`t really an option.
The idea we have is to shorten the filename by removing the UUID, in the best case while generating the file, if that isn`t possible by renaming it afterwards.
The enviorment can be summarized as following:
- Data Bricks Cluster with Python Notebooks
- Files are stored in an Azure Data Lake
Enivormentdata for the used Python:
- Compiler: GCC 5.4.0 20160609
- Build: ('default', 'Nov 23 2017 16:37:01')
- Implementation: CPython Python Version: 3.5.2
python uuid parquet cpython
asked Mar 27 at 12:23
BossBologneseBossBolognese
84 bronze badges
We store some data in partitioned parquet-files for a client. While that works without problems, the client ran into problems through the 255 character-limit in the filepath from windows. The folderstructure that is used for the partitioned data allows to clearly identify the stored data.
Changing the clients folderstructure isn`t really an option.
The idea we have is to shorten the filename by removing the UUID, in the best case while generating the file, if that isn`t possible by renaming it afterwards.
The enviorment can be summarized as following:
- Data Bricks Cluster with Python Notebooks
- Files are stored in an Azure Data Lake
Enivormentdata for the used Python:
- Compiler: GCC 5.4.0 20160609
- Build: ('default', 'Nov 23 2017 16:37:01')
- Implementation: CPython Python Version: 3.5.2
python uuid parquet cpython
python uuid parquet cpython
asked Mar 27 at 12:23
BossBologneseBossBolognese
84 bronze badges
asked Mar 27 at 12:23
BossBologneseBossBolognese
84 bronze badges
asked Mar 27 at 12:23
BossBologneseBossBolognese
84 bronze badges
asked Mar 27 at 12:23
BossBologneseBossBolognese
84 bronze badges
asked Mar 27 at 12:23
BossBologneseBossBolognese
84 bronze badges
84 bronze badges
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55377137%2fcan-you-remove-the-uuid-in-partitioned-parquet-files%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55377137%2fcan-you-remove-the-uuid-in-partitioned-parquet-files%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
