Styjun

I am quite new to C++ so sorry if I ask something silly, but I found no answer online (only a post that refers to python (Can mmap and gzip collaborate?)), trying to see if is possible to read a .GZ file through mmap() function (following: Fast textfile reading in c++) in order to operate some operations on the file and write it down on another file.
I need to retain only some part of the original rows and columns based on some columns/fields values to later retrieve them and compare with other similar files but from different subjects, in order to extract similarities/differences. The files are very big (up to 10GB .GZ) so I am trying to use fast comparison methods for GZIP files.
It is more a "performance comparison" with other methods. Here is it the code (sorry, it is long and I think awful):

#include <algorithm>
#include <iostream>
#include <cstring>
#include <vector>
#include <typeinfo>

// for mmap:
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>

//for writefile
#include <fstream>

template <int N>
void emptyArray( char (&arrayName) [N] ) 
 std::fill( std::begin( arrayName ), std::end( arrayName ), 0 );


const char* map_file(const char* fname, size_t& length);

int main() 

//prende la dimensione del file da aprire
size_t length;
auto f = map_file("myfile.vcf", length);
auto l = f + length;


uintmax_t m_numLines = 0;

std::vector<int> v0;
std::vector<int> v1;
std::vector<int> v2;

for (int i=1; i<length; i++) 
 //vettore di posizioni # in prima posizione di una linea
 if (f[i] == '#' && f[i-1] == 'n') v0.push_back(i);
 //vettore di nuove linee
 if (f[i] == 'n') v1.push_back(i+1);


std::vector<int> inter;
set_intersection(v0.begin(), v0.end(),
 v1.begin(), v1.end(),
 back_inserter(inter));

v1.erase(set_difference(v1.begin(), v1.end(),
 inter.begin(), inter.end(),
 v1.begin()), v1.end());

v1.pop_back();


char chromArray[3];
char posArray[10];
char refArray[50];
char altArray[50];
char qualityArray[10];
char gtArray[4];
char gqxArray[5];
char dpArray[5];
char adArray[10];

//LOOP per NUM RIGA
//apro loop su vettore NL (non #)
for (int nl =0; nl<v1.size(); nl++) qi < 2) continue;
 if (stoi(gqxValue) < 30) continue;

 std::ofstream myfile ("myRes.txt", std::ios_base::app);
 if (myfile.is_open()) 
 myfile <<
 nl << "t" <<
 chromValue << "-" << posValue << "-" << refValue << "-" << altValue << "t" <<
 gtValue << "t" <<
 gqxValue << "t" <<
 quality << "t" <<
 dpValue << "t" <<
 adValue <<
 "n";
 myfile.close();
 else 
 std::cout << "Unable to open file" << 'n';
 




void handle_error(const char* msg) 
perror(msg);
exit(255);


const char* map_file(const char* fname, size_t& length) 

int fd = open(fname, O_RDONLY);

if (fd == -1)
 handle_error("open");

struct stat sb;
if (fstat(fd, &sb) == -1)
 handle_error("fstat");
length = sb.st_size;

const char* addr = static_cast<const char*>(mmap(NULL, length, PROT_READ, MAP_PRIVATE, fd, 0u));
if (addr == MAP_FAILED)
 handle_error("mmap");

return addr;

Now, I know I can open GZIP file with something like:

#include <fstream>
#include <iostream>
#include <sstream>
#include <boost/iostreams/filtering_streambuf.hpp>
#include <boost/iostreams/copy.hpp>
#include <boost/iostreams/filter/gzip.hpp>

//NB: devo linkare a libreria boost zlib in comando: c++ --std=c++11 -L/opt/X11/lib -lboost_iostreams -lz gzread.cpp -o gzread

using namespace std;
using namespace boost::iostreams;

int main()
 ios_base::binary);
filtering_streambuf<input> inbuf; //iniziallizzo filtering_streambuf inbuf
inbuf.push(gzip_decompressor()); //ci metto dentro decompressore GZIP (se file GZIP)
inbuf.push(file); //ci metto dentro file

//Convert streambuf to istream
std::istream instream(&inbuf);
//Iterate lines
std::string line;

string chr;

while(std::getline(instream, line)) 
istringstream iss(line); // string stream della linea
int i = 0;
while (getline(iss, line, t)) // read first part up to comma, ignore the comma (il terzo arfomento di getline gli indica dove fermarsi, se assente si ferma a newline)
if (i == 2) cout << line << "n";
++i;


// copy(inbuf, cout); //copio in stdout

here an example of a file row:

chr1 1246301 . A C 4 OffTarget;LowGQX SNVSB=0.0;SNVHPOL=2;phyloP=1.096;CSQT=1|ACAP3|NM_030649.2|upstream_gene_variant,1|PUSL1|NM_153339.1|missense_variant,1|CPSF3L|NM_001256456.1|downstream_gene_variant GT:GQ:GQX:DP:DPF:AD:PL 0/1:3:0:1:0:0,1:37,3,0

Is there a way to combine them? Or even other approaches if they can be more "performant".

Thanks a lot for any suggestion!

You can read a memory-mapped gzip file with zlib's inflate() functions. (Read the documentation in zlib.h.)

However regardless of whether you read from the file or read from the memory map, you cannot jump around the uncompressed data. The uncompressed data must be processed sequentially, or saved sequentially for later random access processing.