Styjun

I am trying to write a program that reads data from hundreds of YAML files repetitively and stores certain information contained in the files in a table of some sort. The program would essentially parse every YAML file in a given directory and extract the relevant information until every file has been successfully parsed.

An example of the contents of one of the YAML files:

%YAML:1.0
camera_rotation_wrt_base: !!opencv-matrix
 cols: 3
 data: [-0.0159428846, 0.0246045925, 0.999570131, -0.999774337, -0.0144301597, -0.0155909406,
 0.0140403481, -0.999593139, 0.024829099]
 dt: f
 rows: 3
camera_translation_wrt_base: [0.4445618987083435, 0.11700689047574997, 1.5018157958984375]
object_rotation_wrt_base: !!opencv-matrix
 cols: 3
 data: [-0.74130547, -0.0615471229, 0.668339849, 0.669196069, -0.144052029, 0.728989482,
 0.0514085107, 0.987654269, 0.147973642]
 dt: f
 rows: 3
object_rotation_wrt_camera: !!opencv-matrix
 cols: 3
 data: [-0.6565323818253673, 0.1588616842697038, -0.737379262582055, -0.07928983462501557,
 -0.9866892288165471, -0.14197683014545748, -0.7501189077886223, -0.03474571781315458,
 0.6603895951165363]
 dt: f
 rows: 3
object_translation_wrt_base: [1.1534364223480225, 0.05951927974820137, 1.3502429723739624]
object_translation_wrt_camera: [0.04407151401699165, 0.16979082390232392, 0.705698973194305]
template_id: 1965

I would like to be able to store the data key from the object_rotation_wrt_camera key, as well as the object_translation_wrt_camera key, in a CSV file, like so:

observation,rotation,translation
1,[-0.53434, 0.023343, .....],[0.54545,0.34344,....] 
2,[-0.52234, 0.3433, .....],[0.65645,0.8787344,....] 
3,[0.32234, 0.6453, .....],[0.622645,0.1787344,....]

In the above table, the observation number pertains to the yaml file, and therefore for each file there is an observation stored in the CSV file for both the rotation and translation variable. (note: the periods used in the table just indicate that the rotation and translation variables continue as they are quite long).

Lastly, I would like to create a final CSV file, that is similar to the one above however instead has all of the rotation and translation values separated (meaning that instead of one column for translation and one for rotation, there are 3 for translation pertaining to the 3 values within the lists of the previous CSV file, and 9 columns pertaining to each of the 9 values within the lists of the previous CSV file), like so:

observation,tran1,tran2,tran3,rot1,rot2,rot3,rot4,rot5,rot6,rot7,rot8,rot9
1,-0.545434,4.54545,0.343434,.............................................
2,-0.4543,3.3434,0.3534,..................................................

I stored your example in two differently files with .yaml endings and then ran:

import sys
from pathlib import Path
import csv
import ruamel.yaml

result = [['observation', 'rotation', 'translation']]
flatres = ["observation,tran1,tran2,tran3,rot1,rot2,rot3,rot4,rot5,rot6,rot7,rot8,rot9".split(',')]
yaml = ruamel.yaml.YAML()

for idx, file_name in enumerate(Path('.').glob('*.yaml')):
 txt = file_name.read_text()
 if txt.startswith('%YAML:1.0'):
 txt = txt.replace('%YAML:1.0', "", 1).lstrip()
 data = yaml.load(txt)
 result.append([
 idx+1,
 data['object_rotation_wrt_camera']['data'],
 data['object_translation_wrt_camera'],
 ])
 row = [idx+1]
 row.extend(data['object_translation_wrt_camera'])
 row.extend(data['object_rotation_wrt_camera']['data'])
 flatres.append(row)

writer = csv.writer(sys.stdout)
writer.writerows(result)
print('---------')
writer = csv.writer(sys.stdout)
writer.writerows(flatres)

which gives:

observation,rotation,translation
1,"[-0.6565323818253673, 0.1588616842697038, -0.737379262582055, -0.07928983462501557, -0.9866892288165471, -0.14197683014545748, -0.7501189077886223, -0.03474571781315458, 0.6603895951165363]","[0.04407151401699165, 0.16979082390232392, 0.705698973194305]"
2,"[-0.6565323818253673, 0.1588616842697038, -0.737379262582055, -0.07928983462501557, -0.9866892288165471, -0.14197683014545748, -0.7501189077886223, -0.03474571781315458, 0.6603895951165363]","[0.04407151401699165, 0.16979082390232392, 0.705698973194305]"
---------
observation,tran1,tran2,tran3,rot1,rot2,rot3,rot4,rot5,rot6,rot7,rot8,rot9
1,0.04407151401699165,0.16979082390232392,0.705698973194305,-0.6565323818253673,0.1588616842697038,-0.737379262582055,-0.07928983462501557,-0.9866892288165471,-0.14197683014545748,-0.7501189077886223,-0.03474571781315458,0.6603895951165363
2,0.04407151401699165,0.16979082390232392,0.705698973194305,-0.6565323818253673,0.1588616842697038,-0.737379262582055,-0.07928983462501557,-0.9866892288165471,-0.14197683014545748,-0.7501189077886223,-0.03474571781315458,0.6603895951165363

Because there are commas in the sequences that you want to have
stored, those entries need to be quoted and Python's CSV writer does
that automatically (otherwise the second and following rows of the CSV
would have many more than three elements).

YAML 1.0 was superseded, by YAML 1.1, in 2005 and I have not paid
special attention to the differences between those versions apart from
the directive. YAML 1.2 has been the YAML standard since 2009.
ruamel.yaml only supports files that have explicit YAML 1.2 (or 1.1)
directives, that is why the %YAML:1.0 directive needs to be
explicitly removed from these files.

You might run into trouble if you have any "old-style" octal integers
in your files and in some other cases not shown in your input example.