Styjun

How to creep the reader out with what seems like a normal person?

Upright [...] in italics quotation

How deep to place a deadman anchor for a slackline?

Counterexample: a pair of linearly ordered sets that are isomorphic to subsets of the other, but not isomorphic between them

Why do Ichisongas hate elephants and hippos?

How to determine the actual or "true" resolution of a digital photograph?

Why is current rating for multicore cable lower than single core with the same cross section?

Lock in SQL Server and Oracle

Illegal assignment from SObject to Contact

Why didn't this hurt this character as badly?

Is GOCE a satellite or aircraft?

TikZ how to make supply and demand arrows for nodes?

What is the strongest case that can be made in favour of the UK regaining some control over fishing policy after Brexit?

Why is the origin of “threshold” uncertain?

What does YCWCYODFTRFDTY mean?

You look catfish vs You look like a catfish

Does jamais mean always or never in this context?

How can I record the screen and the rear camera on an iPhone simultaneously?

Does a creature that is immune to a condition still make a saving throw?

Unexpected email from Yorkshire Bank

Why do computer-science majors learn calculus?

Stateful vs non-stateful app

How to back up a running remote server?

Is it possible to measure lightning discharges as Nikola Tesla?

What is Pandas Rolling Mean Algorithm with 1m Data and Time Offset 1hr

What is the meaning of a single and a double underscore before an object name?“Large data” work flows using pandasChange data type of columns in PandasRolling mean on pandas data frame timeseriesPandas rolling mean on time seriespandas rolling() function with monthly offsetPandas GroupBy Datetime and Mean using RollingGet mean time/date in pandas datetime seriescalculate rolling mean with lookback period efficiently in pandasRolling Mean with Time Offset Pandas

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;

0

I am trying to figure out the algorithm pandas is using for the rolling mean with time offset because I am getting unexpected results when I compute it manually.

My data is formatted in a data frame with values at 1 minute granularity as follows:

Datetime Values
2018-02-02 21:27:00 3175.068017
2018-02-02 21:28:00 3163.183960
2018-02-02 21:29:00 3175.972021
2018-02-02 21:30:00 3188.535987
2018-02-02 21:31:00 3192.447974

The dtypes for the columns are:

Datetime is datetime64[ns]
Values is float64

Sometimes the time series is irregular (a minute - or more - is skipped) so I am trying to use the time offset for the rolling mean. I want 1hr rolling mean and the command I use is:

df.rolling('H').mean()

My understanding is that this defaults to min_periods=1. I tried with 1 minute rolling mean already and I got back what I expected. But for the 1hr, I don't get what I expect but I don't know what algorithm it is using either.

So with min_periods=1, I am expecting the calculation to be:

Datetime Avg Interval
2018-02-02 21:27:00 3216.71147 [21:27-22:26]
2018-02-02 21:28:00 3217.57497 [21:28-22:27]
2018-02-02 21:29:00 3219.08083 [21:29-22:28]

Instead pandas gives me:

Datetime Avg
2018-02-02 21:27:00 3175.068017 # Same as Raw
2018-02-02 21:28:00 3169.125989
2018-02-02 21:29:00 3171.408000
2018-02-02 21:30:00 3175.689996

I thought maybe it's assuming 21:00-21:59 but the average of the values in that window is 3190.77915. I am basically just trying to determine what pandas is actually doing for my 1m data at a 1H rolling mean granularity because it's not what I expect.

python pandas datetime moving-average rolling-computation

share|improve this question

asked Mar 22 at 19:08

coopercooper

10512

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  It doesn't look forward, it looks backward. SO your mean for 21:28 is 20:28-21:28
  
  – RafaelC
  Mar 22 at 19:11
  
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  @RafaelC Wow, I feel silly but thank you SOOOOO much!
  
  – cooper
  Mar 22 at 19:14
  

  
  
  
  

add a comment | 

0

I am trying to figure out the algorithm pandas is using for the rolling mean with time offset because I am getting unexpected results when I compute it manually.

My data is formatted in a data frame with values at 1 minute granularity as follows:

Datetime Values
2018-02-02 21:27:00 3175.068017
2018-02-02 21:28:00 3163.183960
2018-02-02 21:29:00 3175.972021
2018-02-02 21:30:00 3188.535987
2018-02-02 21:31:00 3192.447974

The dtypes for the columns are:

Datetime is datetime64[ns]
Values is float64

Sometimes the time series is irregular (a minute - or more - is skipped) so I am trying to use the time offset for the rolling mean. I want 1hr rolling mean and the command I use is:

df.rolling('H').mean()

My understanding is that this defaults to min_periods=1. I tried with 1 minute rolling mean already and I got back what I expected. But for the 1hr, I don't get what I expect but I don't know what algorithm it is using either.

So with min_periods=1, I am expecting the calculation to be:

Datetime Avg Interval
2018-02-02 21:27:00 3216.71147 [21:27-22:26]
2018-02-02 21:28:00 3217.57497 [21:28-22:27]
2018-02-02 21:29:00 3219.08083 [21:29-22:28]

Instead pandas gives me:

Datetime Avg
2018-02-02 21:27:00 3175.068017 # Same as Raw
2018-02-02 21:28:00 3169.125989
2018-02-02 21:29:00 3171.408000
2018-02-02 21:30:00 3175.689996

I thought maybe it's assuming 21:00-21:59 but the average of the values in that window is 3190.77915. I am basically just trying to determine what pandas is actually doing for my 1m data at a 1H rolling mean granularity because it's not what I expect.

python pandas datetime moving-average rolling-computation

share|improve this question

asked Mar 22 at 19:08

coopercooper

10512

• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  It doesn't look forward, it looks backward. SO your mean for 21:28 is 20:28-21:28
  
  – RafaelC
  Mar 22 at 19:11
  
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  @RafaelC Wow, I feel silly but thank you SOOOOO much!
  
  – cooper
  Mar 22 at 19:14
  

  
  
  
  

add a comment | 

I am trying to figure out the algorithm pandas is using for the rolling mean with time offset because I am getting unexpected results when I compute it manually.

My data is formatted in a data frame with values at 1 minute granularity as follows:

Datetime Values
2018-02-02 21:27:00 3175.068017
2018-02-02 21:28:00 3163.183960
2018-02-02 21:29:00 3175.972021
2018-02-02 21:30:00 3188.535987
2018-02-02 21:31:00 3192.447974

The dtypes for the columns are:

Datetime is datetime64[ns]
Values is float64

Sometimes the time series is irregular (a minute - or more - is skipped) so I am trying to use the time offset for the rolling mean. I want 1hr rolling mean and the command I use is:

df.rolling('H').mean()

My understanding is that this defaults to min_periods=1. I tried with 1 minute rolling mean already and I got back what I expected. But for the 1hr, I don't get what I expect but I don't know what algorithm it is using either.

So with min_periods=1, I am expecting the calculation to be:

Datetime Avg Interval
2018-02-02 21:27:00 3216.71147 [21:27-22:26]
2018-02-02 21:28:00 3217.57497 [21:28-22:27]
2018-02-02 21:29:00 3219.08083 [21:29-22:28]

Instead pandas gives me:

Datetime Avg
2018-02-02 21:27:00 3175.068017 # Same as Raw
2018-02-02 21:28:00 3169.125989
2018-02-02 21:29:00 3171.408000
2018-02-02 21:30:00 3175.689996

I thought maybe it's assuming 21:00-21:59 but the average of the values in that window is 3190.77915. I am basically just trying to determine what pandas is actually doing for my 1m data at a 1H rolling mean granularity because it's not what I expect.

python pandas datetime moving-average rolling-computation

share|improve this question

asked Mar 22 at 19:08

coopercooper

10512

Datetime Values
2018-02-02 21:27:00 3175.068017
2018-02-02 21:28:00 3163.183960
2018-02-02 21:29:00 3175.972021
2018-02-02 21:30:00 3188.535987
2018-02-02 21:31:00 3192.447974

Datetime is datetime64[ns]
Values is float64

df.rolling('H').mean()

Datetime Avg Interval
2018-02-02 21:27:00 3216.71147 [21:27-22:26]
2018-02-02 21:28:00 3217.57497 [21:28-22:27]
2018-02-02 21:29:00 3219.08083 [21:29-22:28]

Datetime Avg
2018-02-02 21:27:00 3175.068017 # Same as Raw
2018-02-02 21:28:00 3169.125989
2018-02-02 21:29:00 3171.408000
2018-02-02 21:30:00 3175.689996

share|improve this question

asked Mar 22 at 19:08

coopercooper

10512

share|improve this question

asked Mar 22 at 19:08

coopercooper

10512

asked Mar 22 at 19:08

coopercooper

10512

asked Mar 22 at 19:08

coopercooper

10512