Styjun

I am very new to python and pandas and my limited experience has led me to come up with an inefficient solution making my code too slow.

I have some data corresponding to stock market prices.
Sampling is random at nanosecond level.

What I am trying to achieve is transform to a new data-set with fixed sampling rate.

I am transforming my data-set as follows:

• I'm setting a time_delta as a static time step of 0.5 seconds


• I'm dropping reccords corresponding to the same nanosecond


• I'm generating timestamps from my start_time to the calculated end_time
  


• I'm iterating through my original dataframe copying (and duplicating when needed) the last known record in my time_delta for each step to a new dataframe.

I believe that my issue is probably that I am appending the records one-by-one to my new dataframe, however I haven't been able to figure out a way utilizing pandas built-ins to optimize my code.

Runtime is currently ~4min for a day's data (turning ~30K samples to 57600) when executing on Google Colab.

I've also tested locally without any improvement.

# ====================================================================
# Rate Re-Definition
# ====================================================================

SAMPLES_PER_SECOND = 2
dt = 1000000000 / SAMPLES_PER_SECOND # Time delta in nanoseconds
SECONDS_IN_WORK_DAY = 28800 # 60 seconds * 60 minutes * 8 hours
TOTAL_SAMPLES = SECONDS_IN_WORK_DAY * SAMPLES_PER_SECOND
SAMPLING_PERIOD = dt * TOTAL_SAMPLES

start_of_day_timestamp = ceil_to_minute(df['TimeStamp'].iloc[0])
end_of_day_timestamp = start_of_day_timestamp + SAMPLING_PERIOD

fixed_timestamps = np.arange(start_of_day_timestamp,
 end_of_day_timestamp,
 dt,
 dtype=np.uint64
 )


# ====================================================================
# Drop records corresponding to the same timestamps
# ====================================================================

df1 = df.drop_duplicates(subset='TimeStamp', keep="last")


# ====================================================================
# Construct new dataframe
# ====================================================================

df2 = df1.iloc[0:1]
index_bounds_limit = df1.shape[0] - 1
index = 0

for i in tqdm(range(1, TOTAL_SAMPLES), desc="Constructing fixed sampling rate records... "):
 while index < index_bounds_limit and df1['TimeStamp'].iloc[index] < fixed_timestamps[i]:
 index += 1 
 df2 = df2.append(df1.iloc[index], ignore_index=True)

df2['TimeStamp'] = fixed_timestamps

I need to reduce the time as much as possible (while maintaining readability/maintainability, no need to use "hacks").

I would appreciate any help and pointers towards the right direction.

Thanks in advance