Styjun

Pressure inside an infinite ocean?

Did we get closer to another plane than we were supposed to, or was the pilot just protecting our delicate sensibilities?

29er Road Tire?

Find the cheapest shipping option based on item weight

Why wasn't the Night King naked in S08E03?

Adding command shortcuts to bin

Can my company stop me from working overtime?

Decoupling cap routing on a 4 layer PCB

How I can I roll a number of non-digital dice to get a random number between 1 and 150?

Floor of Riemann zeta function

When two pilots are required for a private aircraft, is it a requirement for the PIC to be ATPL?

Should I dumb down my writing in a foreign country?

Are the Night's Watch still required?

Why does this derived table improve performance?

A factorization game

In Stroustrup's example, what does this colon mean in `return 1 : 2`? It's not a label or ternary operator

What was the first story to feature the plot "the monsters were human all along"?

Out of scope work duties and resignation

Word meaning as function of the composition of its phonemes

What does "Managed by Windows" do in the Power options for network connection?

Word for Food that's Gone 'Bad', but is Still Edible?

Can you Ready a Bard spell to release it after using Battle Magic?

Frequency of specific viral sequence in .BAM or .fastq

Where is the documentation for this ex command?

pandas.read_csv() can apply different date formats within the same column! Is it a known bug? How to fix it?

What format to export pandas dataframe while retaining data types? Not CSV; Sqlite? Parquet?How to print a date in a regular format?Where can I find documentation on formatting a date in JavaScript?How to format a JavaScript dateHow can I print literal curly-brace characters in python string and also use .format on it?Parse dates when year month day and hour are in separate columns using pandas in pythonWhich is the fastest way to extract day, month and year from a given date?How to combine dates and hours column into one index column in a pandas series?How to read_excel with a dayfirst condition?how to convert a string type to date formatHow to most efficiently split a date represented as a string in Pandas column?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

2

I have realised that, unless the format of a date column is declared explicitly or semi-explicitly (with dayfirst), pandas can apply different date formats to the same column, when reading a csv file! One row could be dd/mm/yyyy and another row in the same column mm/dd/yyyy!
Insane doesn't even come close to describing it! Is it a known bug?

To demonstrate: the script below creates a very simple table with the dates from January 1st to the 31st, in the dd/mm/yyyy format, saves it to a csv file, then reads back the csv.

I then use pandas.DatetimeIndex to extract the day.
Well, the day is 1 for the first 12 days (when month and day were both < 13), and 13 14 etc afterwards. How on earth is this possible?

The only way I have found to fix this is to declare the date format, either explicitly or just with dayfirst=True. But it's a pain because it means I must declare the date format even when I import csv with the best-formatted dates ever! Is there a simpler way?

This happens to me with pandas 0.23.4 and Python 3.7.1 on Windows 10

import numpy as np
import pandas as pd
df=pd.DataFrame()
df['day'] =np.arange(1,32)
df['day']=df['day'].apply(lambda x: ":0>2d".format(x) )
df['month']='01'
df['year']='2018'
df['date']=df['day']+'/'+df['month']+'/'+df['year']
df.to_csv('mydates.csv', index=False)

#same results whether you use parse_dates or not
imp = pd.read_csv('mydates.csv',parse_dates=['date'])
imp['day extracted']=pd.DatetimeIndex(imp['date']).day
print(imp['day extracted'])

python pandas csv date

share|improve this question

asked Mar 22 at 23:51

Pythonista anonymousPythonista anonymous

1,50662657

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I have also come across this problem in the past, but I don't remember how I fixed. I believe it is worth raising an issue
  
  – RafaelC
  Mar 23 at 2:57
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This has apparently been a known issue for almost 3 years: github.com/pandas-dev/pandas/issues/12585 I am speechless. Different date formats in the same field...
  
  – Pythonista anonymous
  Mar 23 at 16:56
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  AFAIK the one and only way to fix this is to declare the date formats explicitly. See the discussion at: github.com/pandas-dev/pandas/issues/… This is a HUGE issue which has the potential of having messed up years of work by tons of users.
  
  – Pythonista anonymous
  Mar 24 at 11:23
  

  
  
  
  

add a comment | 

2

I have realised that, unless the format of a date column is declared explicitly or semi-explicitly (with dayfirst), pandas can apply different date formats to the same column, when reading a csv file! One row could be dd/mm/yyyy and another row in the same column mm/dd/yyyy!
Insane doesn't even come close to describing it! Is it a known bug?

To demonstrate: the script below creates a very simple table with the dates from January 1st to the 31st, in the dd/mm/yyyy format, saves it to a csv file, then reads back the csv.

I then use pandas.DatetimeIndex to extract the day.
Well, the day is 1 for the first 12 days (when month and day were both < 13), and 13 14 etc afterwards. How on earth is this possible?

The only way I have found to fix this is to declare the date format, either explicitly or just with dayfirst=True. But it's a pain because it means I must declare the date format even when I import csv with the best-formatted dates ever! Is there a simpler way?

This happens to me with pandas 0.23.4 and Python 3.7.1 on Windows 10

import numpy as np
import pandas as pd
df=pd.DataFrame()
df['day'] =np.arange(1,32)
df['day']=df['day'].apply(lambda x: ":0>2d".format(x) )
df['month']='01'
df['year']='2018'
df['date']=df['day']+'/'+df['month']+'/'+df['year']
df.to_csv('mydates.csv', index=False)

#same results whether you use parse_dates or not
imp = pd.read_csv('mydates.csv',parse_dates=['date'])
imp['day extracted']=pd.DatetimeIndex(imp['date']).day
print(imp['day extracted'])

python pandas csv date

share|improve this question

asked Mar 22 at 23:51

Pythonista anonymousPythonista anonymous

1,50662657

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I have also come across this problem in the past, but I don't remember how I fixed. I believe it is worth raising an issue
  
  – RafaelC
  Mar 23 at 2:57
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  This has apparently been a known issue for almost 3 years: github.com/pandas-dev/pandas/issues/12585 I am speechless. Different date formats in the same field...
  
  – Pythonista anonymous
  Mar 23 at 16:56
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  AFAIK the one and only way to fix this is to declare the date formats explicitly. See the discussion at: github.com/pandas-dev/pandas/issues/… This is a HUGE issue which has the potential of having messed up years of work by tons of users.
  
  – Pythonista anonymous
  Mar 24 at 11:23
  

  
  
  
  

add a comment | 

I have realised that, unless the format of a date column is declared explicitly or semi-explicitly (with dayfirst), pandas can apply different date formats to the same column, when reading a csv file! One row could be dd/mm/yyyy and another row in the same column mm/dd/yyyy!
Insane doesn't even come close to describing it! Is it a known bug?

To demonstrate: the script below creates a very simple table with the dates from January 1st to the 31st, in the dd/mm/yyyy format, saves it to a csv file, then reads back the csv.

I then use pandas.DatetimeIndex to extract the day.
Well, the day is 1 for the first 12 days (when month and day were both < 13), and 13 14 etc afterwards. How on earth is this possible?

The only way I have found to fix this is to declare the date format, either explicitly or just with dayfirst=True. But it's a pain because it means I must declare the date format even when I import csv with the best-formatted dates ever! Is there a simpler way?

This happens to me with pandas 0.23.4 and Python 3.7.1 on Windows 10

import numpy as np
import pandas as pd
df=pd.DataFrame()
df['day'] =np.arange(1,32)
df['day']=df['day'].apply(lambda x: ":0>2d".format(x) )
df['month']='01'
df['year']='2018'
df['date']=df['day']+'/'+df['month']+'/'+df['year']
df.to_csv('mydates.csv', index=False)

#same results whether you use parse_dates or not
imp = pd.read_csv('mydates.csv',parse_dates=['date'])
imp['day extracted']=pd.DatetimeIndex(imp['date']).day
print(imp['day extracted'])

python pandas csv date

share|improve this question

asked Mar 22 at 23:51

Pythonista anonymousPythonista anonymous

1,50662657

import numpy as np
import pandas as pd
df=pd.DataFrame()
df['day'] =np.arange(1,32)
df['day']=df['day'].apply(lambda x: ":0>2d".format(x) )
df['month']='01'
df['year']='2018'
df['date']=df['day']+'/'+df['month']+'/'+df['year']
df.to_csv('mydates.csv', index=False)

#same results whether you use parse_dates or not
imp = pd.read_csv('mydates.csv',parse_dates=['date'])
imp['day extracted']=pd.DatetimeIndex(imp['date']).day
print(imp['day extracted'])

share|improve this question

asked Mar 22 at 23:51

Pythonista anonymousPythonista anonymous

1,50662657

share|improve this question

asked Mar 22 at 23:51

Pythonista anonymousPythonista anonymous

1,50662657

asked Mar 22 at 23:51

Pythonista anonymousPythonista anonymous

1,50662657

asked Mar 22 at 23:51

Pythonista anonymousPythonista anonymous

1,50662657