Styjun

I have CSV containing 5000 rows, every few hundred CSV lines there is a repeating section.

What is the most efficient option to divide this file into several different ones?

File looks like

Header1
number of Samples1
Content1
a1, aa1, aaa1
b1, bb1, bbb1
....
Header2
number of Samples2
Content2
a2, aa2, aaa2
b2, bb2, bbb2
....
Header3
number of Samples3
Content3
a3, aa3, aaa3
b3, bb3, bbb3

I need to split it into a few files by Header. And I have no idea how can I do that. I write the whole script to process some biological stuff, but one of the files types (above) generates problems because it is several files in one. And the script does not want to work with it.

I've read a lot about the splitting files, but I've found nothing about the separation after repeating values in pandas.

In this case, it would be 3 files (but the number of these files in files varies)

I found a bit better solution than break statements, as I suggested in comment:

You can create the result list and store each chunk data in separate element of list (in dict, for example). If you read non-Header line, you can guarantee, that the line you just read is related to the current chunk of data. And the current chunk of data is the last element in result list, so you can just modify it. If you read the Header line, you just append the new element to the result and start to write new chunk data into it.

If the size of content is constant, you can use the itertools.cycle iterator that will "codify" your parsing process:

from itertools import cycle

text1 = """Header1
number of Samples1
Content1
a1, aa1, aaa1
b1, bb1, bbb1
Header2
number of Samples2
Content2
a2, aa2, aaa2
b2, bb2, bbb2"""
size = 5
iterator = cycle(range(size))
result = []
for line in text1.split('n'):
 i = next(iterator)
 if i == 0:
 result.append('header': line)
 elif i == 1:
 result[-1]['num_of_samples'] = line
 elif i == 2:
 result[-1]['content_header'] = line
 elif i == 3:
 result[-1]['content'] = [line.split(', ')]
 else:
 result[-1]['content'].append(line.split(', '))

If you don't know the size of content, you should parse each line, check its type and construct your data manually:

text2 = """Header1
number of Samples1
Content1
a1, aa1, aaa1
b1, bb1, bbb1
b1, bb1, bbb1
Header2
number of Samples2
Content2
b2, bb2, bbb2
Header3
number of Samples3
Content3
a3, aa3, aaa3
b3, bb3, bbb3"""
result = []
i = 0
for line in text2.split('n'):
 if line.startswith('Header'): # Your condition for headers
 result.append('header': line)
 elif line.startswith('number'): # Your condition for number of samples
 result[-1]['num_of_samples'] = line
 elif line.startswith('Content'): # Your condition for content headers
 result[-1]['content_header'] = line
 else:
 if 'content' not in result[-1]: # We don't know is the content list created
 result[-1]['content'] = [line.split(', ')]
 else:
 result[-1]['content'].append(line.split(', '))