Styjun

I am working on a project that involves 470 stats kept over a period of 40 years. My data is structured so that each year is a folder and its 470 corresponding stats (as .csv files) are kept inside that folder. Each of these files contains an ascending or descending column of numbers (inside the same .csv file) of which I care about. I need to know what column number this is for every .csv file.

Here is my problem. The information file of which I have been given that explains to me each of these stats is mislabeled. It tells me that the meaningful data is in column 4 for every .csv file (data_col = 4 for all attributes); however, this is only correct for 80-90% of the files from my observations.

I want to batch import column 4 from each file into one large data set (470 columns; each representing a stat) so I can see if the data in that column is ascending or descending. If it is not, I can manually open that file and find it myself. I can then modify my information file so I know which columns to run my analysis on later.

The alternative to this is opening and closing 470 files for each year (or 40 times) and looking to see if the data is in column 4 as stated.

How do I batch import a single column from multiple .csv files into one dataset in R?

This will get all the CSV files in a directory and then read them into a single data.frame only keeping column 4 (or whatever columns you want to keep), and also put the file name in each row if you want to find errors in the input

library(tidyverse) # get useful functions
files <- list.files(path = '/d$/temp/csv', 
 pattern = ".*csv",
 full.names = TRUE
)
all_data <- map_dfr(files, ~
 read_csv(.x)[, 4] %>% # keep column 4
 mutate(file = .x) # add file name
)
print(all_data[1:10,])

col4 file
4 /d$/temp/csv/Document1 - Copy (2).csv
4 /d$/temp/csv/Document1 - Copy (2).csv
4 /d$/temp/csv/Document1 - Copy (2).csv
4 /d$/temp/csv/Document1 - Copy (2).csv
4 /d$/temp/csv/Document1 - Copy (2).csv
4 /d$/temp/csv/Document1 - Copy (2).csv
4 /d$/temp/csv/Document1 - Copy (2).csv
4 /d$/temp/csv/Document1 - Copy (2).csv
4 /d$/temp/csv/Document1 - Copy (2).csv
4 /d$/temp/csv/Document1 - Copy (2).csv

The CSV files looked like this:

col1,col2,col3,col4,col5,col6
1,2,3,4,5,6
1,2,3,4,5,6