Best way to merge/concatenate/join two DataFrames with duplicate columns, but the different Datetime indices?Pandas: Concatenate dataframe and keep duplicate indicesPandas - merge two DataFrames with Identical Column NamesPandas merge two dataframes with different columnsmerge pandas dataframe with key duplicatesConcatenate/Merge/Join two different Dataframes PandasMerge duplicate columns PandasDataFrame merging with ordered indices and different columnsHow to merge pandas dataframe and filter duplicate content in different column?Pandas merge or join in smaller dataframeMerge columns of two dataframes without duplicate rows
Was it ever possible to target a zone?
What are some interesting features that are common cross-linguistically but don't exist in English?
Is for(( ... )) ... ; a valid shell syntax? In which shells?
How many String objects would be created when concatenating multiple Strings?
How many US airports have 4 or more parallel runways?
Improving Performance of an XY Monte Carlo
Why is there so little discussion / research on the philosophy of precision?
Could George I (of Great Britain) speak English?
Why isn't "I've" a proper response?
How can I unambiguously ask for a new user's "Display Name"?
“T” in subscript in formulas
Why are non-collision-resistant hash functions considered insecure for signing self-generated information
What is this symbol: semicircles facing each other?
Did anyone try to find the little box that held Professor Moriarty and his wife after the crash?
Understanding Parallelize methods
What is the difference between Major and Minor Bug?
How do I make my image comply with the requirements of this photography competition?
What to say to a student who has failed?
'Us students' - Does this apposition need a comma?
Asymmetric table
French abbreviation for comparing two items ("vs")
How do the Etherealness and Banishment spells interact?
Why do banks “park” their money at the European Central Bank?
Why did Khan ask Admiral James T. Kirk about Project Genesis?
Best way to merge/concatenate/join two DataFrames with duplicate columns, but the different Datetime indices?
Pandas: Concatenate dataframe and keep duplicate indicesPandas - merge two DataFrames with Identical Column NamesPandas merge two dataframes with different columnsmerge pandas dataframe with key duplicatesConcatenate/Merge/Join two different Dataframes PandasMerge duplicate columns PandasDataFrame merging with ordered indices and different columnsHow to merge pandas dataframe and filter duplicate content in different column?Pandas merge or join in smaller dataframeMerge columns of two dataframes without duplicate rows
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty margin-bottom:0;
2
I have two dataframes, one with past data. The other with a prediction. I would like to merge them so that there are no duplicate columns.
My code looks like this:
Past =
X RealData
2019-03-27 12:30:00 8.295 True
2019-03-27 13:00:00 7.707 True
2019-03-27 13:30:00 7.518 True
2019-03-27 14:00:00 7.518 True
2019-03-27 14:30:00 7.518 True
2019-03-27 15:00:00 7.455 True
2019-03-27 15:30:00 7.518 True
2019-03-27 16:00:00 20.244 True
2019-03-27 16:30:00 20.895 True
2019-03-27 17:00:00 21.630 True
2019-03-27 17:30:00 24.360 True
2019-03-27 18:00:00 24.591 True
2019-03-27 18:30:00 26.460 True
2019-03-27 19:00:00 14.280 True
2019-03-27 19:30:00 12.180 True
2019-03-27 20:00:00 11.550 True
2019-03-27 20:30:00 9.051 True
2019-03-27 21:00:00 8.673 True
2019-03-27 21:30:00 7.791 True
Future =
X RealData
2019-03-27 22:30:00 8.450913 False
2019-03-27 23:00:00 8.494944 False
2019-03-27 23:30:00 9.058649 False
2019-03-28 00:00:00 22.055525 False
2019-03-28 00:30:00 23.344284 False
2019-03-28 01:00:00 24.793011 False
2019-03-28 01:30:00 26.203117 False
2019-03-28 02:00:00 27.897289 False
2019-03-28 02:30:00 14.187933 False
2019-03-28 03:00:00 14.110393 False
At the moment, I am trying:
past_future = pd.concat([Future, Past], axis=1, sort=True)
And I am getting this:
X RealData X RealData
2019-03-27 12:30:00 8.295 True NaN NaN
2019-03-27 13:00:00 7.707 True NaN NaN
2019-03-27 13:30:00 7.518 True NaN NaN
2019-03-27 14:00:00 7.518 True NaN NaN
2019-03-27 14:30:00 7.518 True NaN NaN
2019-03-27 15:00:00 7.455 True NaN NaN
2019-03-27 15:30:00 7.518 True NaN NaN
2019-03-27 16:00:00 20.244 True NaN NaN
2019-03-27 16:30:00 20.895 True NaN NaN
2019-03-27 17:00:00 21.630 True NaN NaN
2019-03-27 17:30:00 24.360 True NaN NaN
2019-03-27 18:00:00 24.591 True NaN NaN
2019-03-27 18:30:00 26.460 True NaN NaN
2019-03-27 19:00:00 14.280 True NaN NaN
2019-03-27 19:30:00 12.180 True NaN NaN
2019-03-27 20:00:00 11.550 True NaN NaN
2019-03-27 20:30:00 9.051 True NaN NaN
2019-03-27 21:00:00 8.673 True NaN NaN
2019-03-27 21:30:00 7.791 True NaN NaN
2019-03-27 22:30:00 NaN NaN 8.450913 False
2019-03-27 23:00:00 NaN NaN 8.494944 False
2019-03-27 23:30:00 NaN NaN 9.058649 False
2019-03-28 00:00:00 NaN NaN 22.055525 False
2019-03-28 00:30:00 NaN NaN 23.344284 False
2019-03-28 01:00:00 NaN NaN 24.793011 False
2019-03-28 01:30:00 NaN NaN 26.203117 False
2019-03-28 02:00:00 NaN NaN 27.897289 False
2019-03-28 02:30:00 NaN NaN 14.187933 False
2019-03-28 03:00:00 NaN NaN 14.110393 False
My expected output is just two columns:
X RealData
2019-03-27 12:30:00 8.295 True
2019-03-27 13:00:00 7.707 True
2019-03-27 13:30:00 7.518 True
2019-03-27 14:00:00 7.518 True
... ... ...
2019-03-27 22:30:00 8.450913 False
2019-03-27 23:00:00 8.494944 False
2019-03-27 23:30:00 9.058649 False
Any idea how to handle this?
python pandas
edited Mar 28 at 11:41
baduker
1,2325 gold badges12 silver badges23 bronze badges
asked Mar 27 at 15:54
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
add a comment |
2
I have two dataframes, one with past data. The other with a prediction. I would like to merge them so that there are no duplicate columns.
My code looks like this:
Past =
X RealData
2019-03-27 12:30:00 8.295 True
2019-03-27 13:00:00 7.707 True
2019-03-27 13:30:00 7.518 True
2019-03-27 14:00:00 7.518 True
2019-03-27 14:30:00 7.518 True
2019-03-27 15:00:00 7.455 True
2019-03-27 15:30:00 7.518 True
2019-03-27 16:00:00 20.244 True
2019-03-27 16:30:00 20.895 True
2019-03-27 17:00:00 21.630 True
2019-03-27 17:30:00 24.360 True
2019-03-27 18:00:00 24.591 True
2019-03-27 18:30:00 26.460 True
2019-03-27 19:00:00 14.280 True
2019-03-27 19:30:00 12.180 True
2019-03-27 20:00:00 11.550 True
2019-03-27 20:30:00 9.051 True
2019-03-27 21:00:00 8.673 True
2019-03-27 21:30:00 7.791 True
Future =
X RealData
2019-03-27 22:30:00 8.450913 False
2019-03-27 23:00:00 8.494944 False
2019-03-27 23:30:00 9.058649 False
2019-03-28 00:00:00 22.055525 False
2019-03-28 00:30:00 23.344284 False
2019-03-28 01:00:00 24.793011 False
2019-03-28 01:30:00 26.203117 False
2019-03-28 02:00:00 27.897289 False
2019-03-28 02:30:00 14.187933 False
2019-03-28 03:00:00 14.110393 False
At the moment, I am trying:
past_future = pd.concat([Future, Past], axis=1, sort=True)
And I am getting this:
X RealData X RealData
2019-03-27 12:30:00 8.295 True NaN NaN
2019-03-27 13:00:00 7.707 True NaN NaN
2019-03-27 13:30:00 7.518 True NaN NaN
2019-03-27 14:00:00 7.518 True NaN NaN
2019-03-27 14:30:00 7.518 True NaN NaN
2019-03-27 15:00:00 7.455 True NaN NaN
2019-03-27 15:30:00 7.518 True NaN NaN
2019-03-27 16:00:00 20.244 True NaN NaN
2019-03-27 16:30:00 20.895 True NaN NaN
2019-03-27 17:00:00 21.630 True NaN NaN
2019-03-27 17:30:00 24.360 True NaN NaN
2019-03-27 18:00:00 24.591 True NaN NaN
2019-03-27 18:30:00 26.460 True NaN NaN
2019-03-27 19:00:00 14.280 True NaN NaN
2019-03-27 19:30:00 12.180 True NaN NaN
2019-03-27 20:00:00 11.550 True NaN NaN
2019-03-27 20:30:00 9.051 True NaN NaN
2019-03-27 21:00:00 8.673 True NaN NaN
2019-03-27 21:30:00 7.791 True NaN NaN
2019-03-27 22:30:00 NaN NaN 8.450913 False
2019-03-27 23:00:00 NaN NaN 8.494944 False
2019-03-27 23:30:00 NaN NaN 9.058649 False
2019-03-28 00:00:00 NaN NaN 22.055525 False
2019-03-28 00:30:00 NaN NaN 23.344284 False
2019-03-28 01:00:00 NaN NaN 24.793011 False
2019-03-28 01:30:00 NaN NaN 26.203117 False
2019-03-28 02:00:00 NaN NaN 27.897289 False
2019-03-28 02:30:00 NaN NaN 14.187933 False
2019-03-28 03:00:00 NaN NaN 14.110393 False
My expected output is just two columns:
X RealData
2019-03-27 12:30:00 8.295 True
2019-03-27 13:00:00 7.707 True
2019-03-27 13:30:00 7.518 True
2019-03-27 14:00:00 7.518 True
... ... ...
2019-03-27 22:30:00 8.450913 False
2019-03-27 23:00:00 8.494944 False
2019-03-27 23:30:00 9.058649 False
Any idea how to handle this?
python pandas
edited Mar 28 at 11:41
baduker
1,2325 gold badges12 silver badges23 bronze badges
asked Mar 27 at 15:54
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
2
what is your expected output ?
– WeNYoBen
Mar 27 at 15:55
1
pd.concat([Future, Past]).drop_duplicates()?
– anky_91
Mar 27 at 15:55
@anky_91 that has not worked for me, unless there is a kwarg that goes in the bracket that I am missing
– Luka Vlaskalic
Mar 27 at 16:08
you could try something likeoutput = pd.concat([Future.reset_index(), Past.reset_index()], axis=0)then set the index withoutput.set_index('index', inplace=True)
– ags29
Mar 27 at 17:05
add a comment |
2
2
2
I have two dataframes, one with past data. The other with a prediction. I would like to merge them so that there are no duplicate columns.
My code looks like this:
Past =
X RealData
2019-03-27 12:30:00 8.295 True
2019-03-27 13:00:00 7.707 True
2019-03-27 13:30:00 7.518 True
2019-03-27 14:00:00 7.518 True
2019-03-27 14:30:00 7.518 True
2019-03-27 15:00:00 7.455 True
2019-03-27 15:30:00 7.518 True
2019-03-27 16:00:00 20.244 True
2019-03-27 16:30:00 20.895 True
2019-03-27 17:00:00 21.630 True
2019-03-27 17:30:00 24.360 True
2019-03-27 18:00:00 24.591 True
2019-03-27 18:30:00 26.460 True
2019-03-27 19:00:00 14.280 True
2019-03-27 19:30:00 12.180 True
2019-03-27 20:00:00 11.550 True
2019-03-27 20:30:00 9.051 True
2019-03-27 21:00:00 8.673 True
2019-03-27 21:30:00 7.791 True
Future =
X RealData
2019-03-27 22:30:00 8.450913 False
2019-03-27 23:00:00 8.494944 False
2019-03-27 23:30:00 9.058649 False
2019-03-28 00:00:00 22.055525 False
2019-03-28 00:30:00 23.344284 False
2019-03-28 01:00:00 24.793011 False
2019-03-28 01:30:00 26.203117 False
2019-03-28 02:00:00 27.897289 False
2019-03-28 02:30:00 14.187933 False
2019-03-28 03:00:00 14.110393 False
At the moment, I am trying:
past_future = pd.concat([Future, Past], axis=1, sort=True)
And I am getting this:
X RealData X RealData
2019-03-27 12:30:00 8.295 True NaN NaN
2019-03-27 13:00:00 7.707 True NaN NaN
2019-03-27 13:30:00 7.518 True NaN NaN
2019-03-27 14:00:00 7.518 True NaN NaN
2019-03-27 14:30:00 7.518 True NaN NaN
2019-03-27 15:00:00 7.455 True NaN NaN
2019-03-27 15:30:00 7.518 True NaN NaN
2019-03-27 16:00:00 20.244 True NaN NaN
2019-03-27 16:30:00 20.895 True NaN NaN
2019-03-27 17:00:00 21.630 True NaN NaN
2019-03-27 17:30:00 24.360 True NaN NaN
2019-03-27 18:00:00 24.591 True NaN NaN
2019-03-27 18:30:00 26.460 True NaN NaN
2019-03-27 19:00:00 14.280 True NaN NaN
2019-03-27 19:30:00 12.180 True NaN NaN
2019-03-27 20:00:00 11.550 True NaN NaN
2019-03-27 20:30:00 9.051 True NaN NaN
2019-03-27 21:00:00 8.673 True NaN NaN
2019-03-27 21:30:00 7.791 True NaN NaN
2019-03-27 22:30:00 NaN NaN 8.450913 False
2019-03-27 23:00:00 NaN NaN 8.494944 False
2019-03-27 23:30:00 NaN NaN 9.058649 False
2019-03-28 00:00:00 NaN NaN 22.055525 False
2019-03-28 00:30:00 NaN NaN 23.344284 False
2019-03-28 01:00:00 NaN NaN 24.793011 False
2019-03-28 01:30:00 NaN NaN 26.203117 False
2019-03-28 02:00:00 NaN NaN 27.897289 False
2019-03-28 02:30:00 NaN NaN 14.187933 False
2019-03-28 03:00:00 NaN NaN 14.110393 False
My expected output is just two columns:
X RealData
2019-03-27 12:30:00 8.295 True
2019-03-27 13:00:00 7.707 True
2019-03-27 13:30:00 7.518 True
2019-03-27 14:00:00 7.518 True
... ... ...
2019-03-27 22:30:00 8.450913 False
2019-03-27 23:00:00 8.494944 False
2019-03-27 23:30:00 9.058649 False
Any idea how to handle this?
python pandas
edited Mar 28 at 11:41
baduker
1,2325 gold badges12 silver badges23 bronze badges
asked Mar 27 at 15:54
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
I have two dataframes, one with past data. The other with a prediction. I would like to merge them so that there are no duplicate columns.
My code looks like this:
Past =
X RealData
2019-03-27 12:30:00 8.295 True
2019-03-27 13:00:00 7.707 True
2019-03-27 13:30:00 7.518 True
2019-03-27 14:00:00 7.518 True
2019-03-27 14:30:00 7.518 True
2019-03-27 15:00:00 7.455 True
2019-03-27 15:30:00 7.518 True
2019-03-27 16:00:00 20.244 True
2019-03-27 16:30:00 20.895 True
2019-03-27 17:00:00 21.630 True
2019-03-27 17:30:00 24.360 True
2019-03-27 18:00:00 24.591 True
2019-03-27 18:30:00 26.460 True
2019-03-27 19:00:00 14.280 True
2019-03-27 19:30:00 12.180 True
2019-03-27 20:00:00 11.550 True
2019-03-27 20:30:00 9.051 True
2019-03-27 21:00:00 8.673 True
2019-03-27 21:30:00 7.791 True
Future =
X RealData
2019-03-27 22:30:00 8.450913 False
2019-03-27 23:00:00 8.494944 False
2019-03-27 23:30:00 9.058649 False
2019-03-28 00:00:00 22.055525 False
2019-03-28 00:30:00 23.344284 False
2019-03-28 01:00:00 24.793011 False
2019-03-28 01:30:00 26.203117 False
2019-03-28 02:00:00 27.897289 False
2019-03-28 02:30:00 14.187933 False
2019-03-28 03:00:00 14.110393 False
At the moment, I am trying:
past_future = pd.concat([Future, Past], axis=1, sort=True)
And I am getting this:
X RealData X RealData
2019-03-27 12:30:00 8.295 True NaN NaN
2019-03-27 13:00:00 7.707 True NaN NaN
2019-03-27 13:30:00 7.518 True NaN NaN
2019-03-27 14:00:00 7.518 True NaN NaN
2019-03-27 14:30:00 7.518 True NaN NaN
2019-03-27 15:00:00 7.455 True NaN NaN
2019-03-27 15:30:00 7.518 True NaN NaN
2019-03-27 16:00:00 20.244 True NaN NaN
2019-03-27 16:30:00 20.895 True NaN NaN
2019-03-27 17:00:00 21.630 True NaN NaN
2019-03-27 17:30:00 24.360 True NaN NaN
2019-03-27 18:00:00 24.591 True NaN NaN
2019-03-27 18:30:00 26.460 True NaN NaN
2019-03-27 19:00:00 14.280 True NaN NaN
2019-03-27 19:30:00 12.180 True NaN NaN
2019-03-27 20:00:00 11.550 True NaN NaN
2019-03-27 20:30:00 9.051 True NaN NaN
2019-03-27 21:00:00 8.673 True NaN NaN
2019-03-27 21:30:00 7.791 True NaN NaN
2019-03-27 22:30:00 NaN NaN 8.450913 False
2019-03-27 23:00:00 NaN NaN 8.494944 False
2019-03-27 23:30:00 NaN NaN 9.058649 False
2019-03-28 00:00:00 NaN NaN 22.055525 False
2019-03-28 00:30:00 NaN NaN 23.344284 False
2019-03-28 01:00:00 NaN NaN 24.793011 False
2019-03-28 01:30:00 NaN NaN 26.203117 False
2019-03-28 02:00:00 NaN NaN 27.897289 False
2019-03-28 02:30:00 NaN NaN 14.187933 False
2019-03-28 03:00:00 NaN NaN 14.110393 False
My expected output is just two columns:
X RealData
2019-03-27 12:30:00 8.295 True
2019-03-27 13:00:00 7.707 True
2019-03-27 13:30:00 7.518 True
2019-03-27 14:00:00 7.518 True
... ... ...
2019-03-27 22:30:00 8.450913 False
2019-03-27 23:00:00 8.494944 False
2019-03-27 23:30:00 9.058649 False
Any idea how to handle this?
python pandas
python pandas
edited Mar 28 at 11:41
baduker
1,2325 gold badges12 silver badges23 bronze badges
asked Mar 27 at 15:54
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
edited Mar 28 at 11:41
baduker
1,2325 gold badges12 silver badges23 bronze badges
asked Mar 27 at 15:54
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
edited Mar 28 at 11:41
baduker
1,2325 gold badges12 silver badges23 bronze badges
edited Mar 28 at 11:41
baduker
1,2325 gold badges12 silver badges23 bronze badges
edited Mar 28 at 11:41
baduker
1,2325 gold badges12 silver badges23 bronze badges
1,2325 gold badges12 silver badges23 bronze badges
asked Mar 27 at 15:54
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
asked Mar 27 at 15:54
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
asked Mar 27 at 15:54
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
1278 bronze badges
2
what is your expected output ?
– WeNYoBen
Mar 27 at 15:55
1
pd.concat([Future, Past]).drop_duplicates()?
– anky_91
Mar 27 at 15:55
@anky_91 that has not worked for me, unless there is a kwarg that goes in the bracket that I am missing
– Luka Vlaskalic
Mar 27 at 16:08
you could try something likeoutput = pd.concat([Future.reset_index(), Past.reset_index()], axis=0)then set the index withoutput.set_index('index', inplace=True)
– ags29
Mar 27 at 17:05
add a comment |
2
what is your expected output ?
– WeNYoBen
Mar 27 at 15:55
1
pd.concat([Future, Past]).drop_duplicates()?
– anky_91
Mar 27 at 15:55
@anky_91 that has not worked for me, unless there is a kwarg that goes in the bracket that I am missing
– Luka Vlaskalic
Mar 27 at 16:08
you could try something likeoutput = pd.concat([Future.reset_index(), Past.reset_index()], axis=0)then set the index withoutput.set_index('index', inplace=True)
– ags29
Mar 27 at 17:05
2
2
what is your expected output ?
– WeNYoBen
Mar 27 at 15:55
what is your expected output ?
– WeNYoBen
Mar 27 at 15:55
1
1
pd.concat([Future, Past]).drop_duplicates() ?– anky_91
Mar 27 at 15:55
pd.concat([Future, Past]).drop_duplicates() ?– anky_91
Mar 27 at 15:55
@anky_91 that has not worked for me, unless there is a kwarg that goes in the bracket that I am missing
– Luka Vlaskalic
Mar 27 at 16:08
@anky_91 that has not worked for me, unless there is a kwarg that goes in the bracket that I am missing
– Luka Vlaskalic
Mar 27 at 16:08
you could try something like
output = pd.concat([Future.reset_index(), Past.reset_index()], axis=0) then set the index with output.set_index('index', inplace=True)– ags29
Mar 27 at 17:05
you could try something like
output = pd.concat([Future.reset_index(), Past.reset_index()], axis=0) then set the index with output.set_index('index', inplace=True)– ags29
Mar 27 at 17:05
add a comment |
2 Answers
2
active
oldest
votes
2
My simple advice - keep everything in the order.
Then everything is easy.
import pandas as pd
df1 = pd.read_csv('c:/4/a1.csv')
df2 = pd.read_csv('c:/4/a2.csv')
df2.dtypes
df1.date = pd.to_datetime(df1.date)
df2.date = pd.to_datetime(df1.date)
df2.dtypes
df1.set_index(df1.date, inplace=True)
df2.set_index(df2.date, inplace=True)
df = df1.append(df2)
df.sort_index()
df.drop_duplicates('date',keep='last', inplace=True)
df
answered Mar 27 at 17:52
Wojciech MoszczyńskiWojciech Moszczyński
1091 silver badge11 bronze badges
add a comment |
0
Just to formalise what ags29 wrote here Best way to merge/concatenate/join two DataFrames with duplicate columns, but the different Datetime indices?
output = pd.concat([Future.reset_index(), Past.reset_index()], axis=0)
output.set_index('index', inplace=True)
While Wojciech Moszczyński's answer is much more thorough, this seems to do the job quite well.
answered Mar 28 at 11:27
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55381471%2fbest-way-to-merge-concatenate-join-two-dataframes-with-duplicate-columns-but-th%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
2
My simple advice - keep everything in the order.
Then everything is easy.
import pandas as pd
df1 = pd.read_csv('c:/4/a1.csv')
df2 = pd.read_csv('c:/4/a2.csv')
df2.dtypes
df1.date = pd.to_datetime(df1.date)
df2.date = pd.to_datetime(df1.date)
df2.dtypes
df1.set_index(df1.date, inplace=True)
df2.set_index(df2.date, inplace=True)
df = df1.append(df2)
df.sort_index()
df.drop_duplicates('date',keep='last', inplace=True)
df
answered Mar 27 at 17:52
Wojciech MoszczyńskiWojciech Moszczyński
1091 silver badge11 bronze badges
add a comment |
2
My simple advice - keep everything in the order.
Then everything is easy.
import pandas as pd
df1 = pd.read_csv('c:/4/a1.csv')
df2 = pd.read_csv('c:/4/a2.csv')
df2.dtypes
df1.date = pd.to_datetime(df1.date)
df2.date = pd.to_datetime(df1.date)
df2.dtypes
df1.set_index(df1.date, inplace=True)
df2.set_index(df2.date, inplace=True)
df = df1.append(df2)
df.sort_index()
df.drop_duplicates('date',keep='last', inplace=True)
df
answered Mar 27 at 17:52
Wojciech MoszczyńskiWojciech Moszczyński
1091 silver badge11 bronze badges
add a comment |
2
2
2
My simple advice - keep everything in the order.
Then everything is easy.
import pandas as pd
df1 = pd.read_csv('c:/4/a1.csv')
df2 = pd.read_csv('c:/4/a2.csv')
df2.dtypes
df1.date = pd.to_datetime(df1.date)
df2.date = pd.to_datetime(df1.date)
df2.dtypes
df1.set_index(df1.date, inplace=True)
df2.set_index(df2.date, inplace=True)
df = df1.append(df2)
df.sort_index()
df.drop_duplicates('date',keep='last', inplace=True)
df
answered Mar 27 at 17:52
Wojciech MoszczyńskiWojciech Moszczyński
1091 silver badge11 bronze badges
My simple advice - keep everything in the order.
Then everything is easy.
import pandas as pd
df1 = pd.read_csv('c:/4/a1.csv')
df2 = pd.read_csv('c:/4/a2.csv')
df2.dtypes
df1.date = pd.to_datetime(df1.date)
df2.date = pd.to_datetime(df1.date)
df2.dtypes
df1.set_index(df1.date, inplace=True)
df2.set_index(df2.date, inplace=True)
df = df1.append(df2)
df.sort_index()
df.drop_duplicates('date',keep='last', inplace=True)
df
answered Mar 27 at 17:52
Wojciech MoszczyńskiWojciech Moszczyński
1091 silver badge11 bronze badges
answered Mar 27 at 17:52
Wojciech MoszczyńskiWojciech Moszczyński
1091 silver badge11 bronze badges
answered Mar 27 at 17:52
Wojciech MoszczyńskiWojciech Moszczyński
1091 silver badge11 bronze badges
answered Mar 27 at 17:52
Wojciech MoszczyńskiWojciech Moszczyński
1091 silver badge11 bronze badges
1091 silver badge11 bronze badges
add a comment |
add a comment |
0
Just to formalise what ags29 wrote here Best way to merge/concatenate/join two DataFrames with duplicate columns, but the different Datetime indices?
output = pd.concat([Future.reset_index(), Past.reset_index()], axis=0)
output.set_index('index', inplace=True)
While Wojciech Moszczyński's answer is much more thorough, this seems to do the job quite well.
answered Mar 28 at 11:27
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
add a comment |
0
Just to formalise what ags29 wrote here Best way to merge/concatenate/join two DataFrames with duplicate columns, but the different Datetime indices?
output = pd.concat([Future.reset_index(), Past.reset_index()], axis=0)
output.set_index('index', inplace=True)
While Wojciech Moszczyński's answer is much more thorough, this seems to do the job quite well.
answered Mar 28 at 11:27
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
add a comment |
0
0
0
Just to formalise what ags29 wrote here Best way to merge/concatenate/join two DataFrames with duplicate columns, but the different Datetime indices?
output = pd.concat([Future.reset_index(), Past.reset_index()], axis=0)
output.set_index('index', inplace=True)
While Wojciech Moszczyński's answer is much more thorough, this seems to do the job quite well.
answered Mar 28 at 11:27
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
Just to formalise what ags29 wrote here Best way to merge/concatenate/join two DataFrames with duplicate columns, but the different Datetime indices?
output = pd.concat([Future.reset_index(), Past.reset_index()], axis=0)
output.set_index('index', inplace=True)
While Wojciech Moszczyński's answer is much more thorough, this seems to do the job quite well.
answered Mar 28 at 11:27
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
answered Mar 28 at 11:27
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
answered Mar 28 at 11:27
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
answered Mar 28 at 11:27
Luka VlaskalicLuka Vlaskalic
1278 bronze badges
1278 bronze badges
add a comment |
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55381471%2fbest-way-to-merge-concatenate-join-two-dataframes-with-duplicate-columns-but-th%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
what is your expected output ?
– WeNYoBen
Mar 27 at 15:55
1
pd.concat([Future, Past]).drop_duplicates()?– anky_91
Mar 27 at 15:55
@anky_91 that has not worked for me, unless there is a kwarg that goes in the bracket that I am missing
– Luka Vlaskalic
Mar 27 at 16:08
you could try something like
output = pd.concat([Future.reset_index(), Past.reset_index()], axis=0)then set the index withoutput.set_index('index', inplace=True)– ags29
Mar 27 at 17:05