Using Haskell's Req library with Scalpel to save files to clientWhat is Haskell's Stream FusionComparing Haskell's Snap and Yesod web frameworksJava's Interface and Haskell's type class: differences and similarities?What are Haskell's strictness points?Choice operator in Haskell's parsec libraryHaskell's ($) is a magic operator?Why does Haskell's “do nothing” function, id, consume tons of memory?What does Haskell's <|> operator do?Modifying formatters in Haskell's formatting libraryRead all links to .gz files into a [String] from a page (Haskell / Scalpel)
German equivalent to "going down the rabbit hole"
LINQ Extension methods MinBy and MaxBy
Is the net torque changed when a partner on a seesaw stands or hangs from her end instead of sitting?
Can UV radiation be safe for the skin?
Understanding data transmission rates over copper wire
What is the practical impact of using System.Random which is not cryptographically random?
How can I improve my formal definitions?
How to number subfigures in Serbian Cyrillic?
Using font to highlight a god's speech in dialogue
How were US credit cards verified in-store in the 1980's?
Don't look at what I did there
How can I store milk for long periods of time?
awk print conditions
Fishing from underwater domes
Deck of Many Things. What happens if you don't declare any number of cards and just start drawing?
Is this statement about a motion being simple harmonic in nature strong?
I was given someone else's visa, stamped in my passport
Can I leave a large suitcase at TPE during a 4-hour layover, and pick it up 4.5 days later when I come back to TPE on my way to Taipei downtown?
Break down the phrase "shitsurei shinakereba naranaindesu"
How to investigate an unknown 1.5GB file named "sudo" in my Linux home directory?
Ideas behind the 8.Bd3 line in the 4.Ng5 Two Knights Defense
What caused the end of cybernetic implants?
Does Q ever actually lie?
A word for the urge to do the opposite
Using Haskell's Req library with Scalpel to save files to client
What is Haskell's Stream FusionComparing Haskell's Snap and Yesod web frameworksJava's Interface and Haskell's type class: differences and similarities?What are Haskell's strictness points?Choice operator in Haskell's parsec libraryHaskell's ($) is a magic operator?Why does Haskell's “do nothing” function, id, consume tons of memory?What does Haskell's <|> operator do?Modifying formatters in Haskell's formatting libraryRead all links to .gz files into a [String] from a page (Haskell / Scalpel)
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
I am 99% of the way there on a Haskell / Scalpel scraper to pull down thousands of .gz files in batches and store them locally on my client.
I am trying to now take the chunked [[String]]s and get them into IO so they can be saved.
I am using import Network.HTTP.Req.
-- This is the argument for the downloadFiles function
chunkLinks :: IO [[String]]
chunkLinks = fmap (chunksOf 10) filterNonGz
-- THIS IS WHAT YOU EXECUTE
downloadFile :: (MonadHttp m) => String -> m ()
downloadFile url = do
-- Pull them into memory
contents <- req GET (https (T.pack url)) NoReqBody bsResponse mempty
-- Get the filename itself
let fileName = head $ reverse $ T.splitOn "/" (T.pack url)
-- Write them to the client (the filename will be the url)
liftIO $ BSS.writeFile (dataDir ++ (T.unpack fileName)) (responseBody contents)
dataDir = "./Data/LODES/"
eatChunks :: (MonadHttp m) => m [()]
eatChunks = do
(wtf :: [[String]]) <- liftIO $ (take (5) <$> chunkLinks)
fmap concat $ liftIO $ mapM (liftIO . mapConcurrently (liftIO . downloadFile)) wtf
I am getting an error on downloadFile in the last line.
The error is:
• Could not deduce (MonadHttp IO)
arising from a use of ‘downloadFile’
from the context: MonadHttp m
bound by the type signature for:
eatChunks :: forall (m :: * -> *). MonadHttp m => m [()]
at /private/var/folders/70/kchtzk4j0hs398f95ywd78x00000gn/T/ghc-mod39229/UsCensusDataLodesScraper39228-2281.hs:195:1-37
Please excuse the liftIOs littered about. They are desperate attempts to get this to work...
Does anyone know what to do next?
For reference, the output of chunkLinks looks like this:
[["https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2002.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2003.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2004.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2005.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2006.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2007.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2008.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2009.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2010.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2011.csv.gz"], ...etc.
haskell
asked Mar 27 at 23:57
reallymemorablereallymemorable
1941 gold badge1 silver badge10 bronze badges
add a comment |
I am 99% of the way there on a Haskell / Scalpel scraper to pull down thousands of .gz files in batches and store them locally on my client.
I am trying to now take the chunked [[String]]s and get them into IO so they can be saved.
I am using import Network.HTTP.Req.
-- This is the argument for the downloadFiles function
chunkLinks :: IO [[String]]
chunkLinks = fmap (chunksOf 10) filterNonGz
-- THIS IS WHAT YOU EXECUTE
downloadFile :: (MonadHttp m) => String -> m ()
downloadFile url = do
-- Pull them into memory
contents <- req GET (https (T.pack url)) NoReqBody bsResponse mempty
-- Get the filename itself
let fileName = head $ reverse $ T.splitOn "/" (T.pack url)
-- Write them to the client (the filename will be the url)
liftIO $ BSS.writeFile (dataDir ++ (T.unpack fileName)) (responseBody contents)
dataDir = "./Data/LODES/"
eatChunks :: (MonadHttp m) => m [()]
eatChunks = do
(wtf :: [[String]]) <- liftIO $ (take (5) <$> chunkLinks)
fmap concat $ liftIO $ mapM (liftIO . mapConcurrently (liftIO . downloadFile)) wtf
I am getting an error on downloadFile in the last line.
The error is:
• Could not deduce (MonadHttp IO)
arising from a use of ‘downloadFile’
from the context: MonadHttp m
bound by the type signature for:
eatChunks :: forall (m :: * -> *). MonadHttp m => m [()]
at /private/var/folders/70/kchtzk4j0hs398f95ywd78x00000gn/T/ghc-mod39229/UsCensusDataLodesScraper39228-2281.hs:195:1-37
Please excuse the liftIOs littered about. They are desperate attempts to get this to work...
Does anyone know what to do next?
For reference, the output of chunkLinks looks like this:
[["https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2002.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2003.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2004.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2005.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2006.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2007.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2008.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2009.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2010.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2011.csv.gz"], ...etc.
haskell
asked Mar 27 at 23:57
reallymemorablereallymemorable
1941 gold badge1 silver badge10 bronze badges
1
The only instance ofMonadHttpisReq. The way to run aReqaction isrunReq. So instead of trying to useliftIOto raise yourdownloadFilefromIOto something, which can't work becausedownloadFileisn't anIOaction, userunReqto lower yourdownloadFilefrom something toIO.
– Daniel Wagner
Mar 28 at 0:32
I tried this, but clearly I am still doing something wrong:fmap concat $ mapM (mapConcurrently (runReq downloadFile)) wtf
– reallymemorable
Mar 28 at 16:23
Trying to use defaultHttpConfig:fmap concat $ mapM (mapConcurrently (runReq . (defaultHttpConfig downloadFile))) wtf
– reallymemorable
Mar 28 at 16:42
add a comment |
I am 99% of the way there on a Haskell / Scalpel scraper to pull down thousands of .gz files in batches and store them locally on my client.
I am trying to now take the chunked [[String]]s and get them into IO so they can be saved.
I am using import Network.HTTP.Req.
-- This is the argument for the downloadFiles function
chunkLinks :: IO [[String]]
chunkLinks = fmap (chunksOf 10) filterNonGz
-- THIS IS WHAT YOU EXECUTE
downloadFile :: (MonadHttp m) => String -> m ()
downloadFile url = do
-- Pull them into memory
contents <- req GET (https (T.pack url)) NoReqBody bsResponse mempty
-- Get the filename itself
let fileName = head $ reverse $ T.splitOn "/" (T.pack url)
-- Write them to the client (the filename will be the url)
liftIO $ BSS.writeFile (dataDir ++ (T.unpack fileName)) (responseBody contents)
dataDir = "./Data/LODES/"
eatChunks :: (MonadHttp m) => m [()]
eatChunks = do
(wtf :: [[String]]) <- liftIO $ (take (5) <$> chunkLinks)
fmap concat $ liftIO $ mapM (liftIO . mapConcurrently (liftIO . downloadFile)) wtf
I am getting an error on downloadFile in the last line.
The error is:
• Could not deduce (MonadHttp IO)
arising from a use of ‘downloadFile’
from the context: MonadHttp m
bound by the type signature for:
eatChunks :: forall (m :: * -> *). MonadHttp m => m [()]
at /private/var/folders/70/kchtzk4j0hs398f95ywd78x00000gn/T/ghc-mod39229/UsCensusDataLodesScraper39228-2281.hs:195:1-37
Please excuse the liftIOs littered about. They are desperate attempts to get this to work...
Does anyone know what to do next?
For reference, the output of chunkLinks looks like this:
[["https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2002.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2003.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2004.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2005.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2006.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2007.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2008.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2009.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2010.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2011.csv.gz"], ...etc.
haskell
asked Mar 27 at 23:57
reallymemorablereallymemorable
1941 gold badge1 silver badge10 bronze badges
I am 99% of the way there on a Haskell / Scalpel scraper to pull down thousands of .gz files in batches and store them locally on my client.
I am trying to now take the chunked [[String]]s and get them into IO so they can be saved.
I am using import Network.HTTP.Req.
-- This is the argument for the downloadFiles function
chunkLinks :: IO [[String]]
chunkLinks = fmap (chunksOf 10) filterNonGz
-- THIS IS WHAT YOU EXECUTE
downloadFile :: (MonadHttp m) => String -> m ()
downloadFile url = do
-- Pull them into memory
contents <- req GET (https (T.pack url)) NoReqBody bsResponse mempty
-- Get the filename itself
let fileName = head $ reverse $ T.splitOn "/" (T.pack url)
-- Write them to the client (the filename will be the url)
liftIO $ BSS.writeFile (dataDir ++ (T.unpack fileName)) (responseBody contents)
dataDir = "./Data/LODES/"
eatChunks :: (MonadHttp m) => m [()]
eatChunks = do
(wtf :: [[String]]) <- liftIO $ (take (5) <$> chunkLinks)
fmap concat $ liftIO $ mapM (liftIO . mapConcurrently (liftIO . downloadFile)) wtf
I am getting an error on downloadFile in the last line.
The error is:
• Could not deduce (MonadHttp IO)
arising from a use of ‘downloadFile’
from the context: MonadHttp m
bound by the type signature for:
eatChunks :: forall (m :: * -> *). MonadHttp m => m [()]
at /private/var/folders/70/kchtzk4j0hs398f95ywd78x00000gn/T/ghc-mod39229/UsCensusDataLodesScraper39228-2281.hs:195:1-37
Please excuse the liftIOs littered about. They are desperate attempts to get this to work...
Does anyone know what to do next?
For reference, the output of chunkLinks looks like this:
[["https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2002.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2003.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2004.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2005.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2006.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2007.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2008.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2009.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2010.csv.gz","https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2011.csv.gz"], ...etc.
haskell
haskell
asked Mar 27 at 23:57
reallymemorablereallymemorable
1941 gold badge1 silver badge10 bronze badges
asked Mar 27 at 23:57
reallymemorablereallymemorable
1941 gold badge1 silver badge10 bronze badges
asked Mar 27 at 23:57
reallymemorablereallymemorable
1941 gold badge1 silver badge10 bronze badges
asked Mar 27 at 23:57
reallymemorablereallymemorable
1941 gold badge1 silver badge10 bronze badges
asked Mar 27 at 23:57
reallymemorablereallymemorable
1941 gold badge1 silver badge10 bronze badges
1941 gold badge1 silver badge10 bronze badges
1
The only instance ofMonadHttpisReq. The way to run aReqaction isrunReq. So instead of trying to useliftIOto raise yourdownloadFilefromIOto something, which can't work becausedownloadFileisn't anIOaction, userunReqto lower yourdownloadFilefrom something toIO.
– Daniel Wagner
Mar 28 at 0:32
I tried this, but clearly I am still doing something wrong:fmap concat $ mapM (mapConcurrently (runReq downloadFile)) wtf
– reallymemorable
Mar 28 at 16:23
Trying to use defaultHttpConfig:fmap concat $ mapM (mapConcurrently (runReq . (defaultHttpConfig downloadFile))) wtf
– reallymemorable
Mar 28 at 16:42
add a comment |
1
The only instance ofMonadHttpisReq. The way to run aReqaction isrunReq. So instead of trying to useliftIOto raise yourdownloadFilefromIOto something, which can't work becausedownloadFileisn't anIOaction, userunReqto lower yourdownloadFilefrom something toIO.
– Daniel Wagner
Mar 28 at 0:32
I tried this, but clearly I am still doing something wrong:fmap concat $ mapM (mapConcurrently (runReq downloadFile)) wtf
– reallymemorable
Mar 28 at 16:23
Trying to use defaultHttpConfig:fmap concat $ mapM (mapConcurrently (runReq . (defaultHttpConfig downloadFile))) wtf
– reallymemorable
Mar 28 at 16:42
1
1
The only instance of
MonadHttp is Req. The way to run a Req action is runReq. So instead of trying to use liftIO to raise your downloadFile from IO to something, which can't work because downloadFile isn't an IO action, use runReq to lower your downloadFile from something to IO.– Daniel Wagner
Mar 28 at 0:32
The only instance of
MonadHttp is Req. The way to run a Req action is runReq. So instead of trying to use liftIO to raise your downloadFile from IO to something, which can't work because downloadFile isn't an IO action, use runReq to lower your downloadFile from something to IO.– Daniel Wagner
Mar 28 at 0:32
I tried this, but clearly I am still doing something wrong:
fmap concat $ mapM (mapConcurrently (runReq downloadFile)) wtf– reallymemorable
Mar 28 at 16:23
I tried this, but clearly I am still doing something wrong:
fmap concat $ mapM (mapConcurrently (runReq downloadFile)) wtf– reallymemorable
Mar 28 at 16:23
Trying to use defaultHttpConfig:
fmap concat $ mapM (mapConcurrently (runReq . (defaultHttpConfig downloadFile))) wtf– reallymemorable
Mar 28 at 16:42
Trying to use defaultHttpConfig:
fmap concat $ mapM (mapConcurrently (runReq . (defaultHttpConfig downloadFile))) wtf– reallymemorable
Mar 28 at 16:42
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55388226%2fusing-haskells-req-library-with-scalpel-to-save-files-to-client%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Is this question similar to what you get asked at work? Learn more about asking and sharing private information with your coworkers using Stack Overflow for Teams.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55388226%2fusing-haskells-req-library-with-scalpel-to-save-files-to-client%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
The only instance of
MonadHttpisReq. The way to run aReqaction isrunReq. So instead of trying to useliftIOto raise yourdownloadFilefromIOto something, which can't work becausedownloadFileisn't anIOaction, userunReqto lower yourdownloadFilefrom something toIO.– Daniel Wagner
Mar 28 at 0:32
I tried this, but clearly I am still doing something wrong:
fmap concat $ mapM (mapConcurrently (runReq downloadFile)) wtf– reallymemorable
Mar 28 at 16:23
Trying to use defaultHttpConfig:
fmap concat $ mapM (mapConcurrently (runReq . (defaultHttpConfig downloadFile))) wtf– reallymemorable
Mar 28 at 16:42