Styjun

I have had some trouble understanding the D3.JS fetch documentation:

My data source is:

20180601 000000;1.168200;1.168240;1.168140;1.168230;0;
20180601 000100;1.168220;1.168230;1.168190;1.168190;0;
20180601 000200;1.168180;1.168180;1.168080;1.168120;0;
20180601 000300;1.168130;1.168160;1.168130;1.168140;0;

where the format is:

%Y%m%d %H%M%S;number1;number2;number3;number4;number5;

My difficulties are:

1. Adding headers to the data


2. Dealing with semicolons as the delimiter instead of commas

1) From what I can work out I need to read the file without parsing it, then join a text string to the beginning of the file then finally parse the data.

d3.text(data.csv, function(error, textString));

var headers = ["date","time","data1","data2"].join("t");

d3.csv.parse(headers + textString);

2) I can use the dsv format and set the delimiter to semicolons?

d3.dsv(";", "text/plain")

The rough code I ended up with is:

var time_parse = d3.timeParse( '%Y%m%d %H%M%S');
var time_format = d3.timeFormat('%H%M');

d3.text(data.csv, function(error, textString)

var headers = ["time;number1;number2;number3;number4;number5;"].join("t")

d3.csv.parse(headers + textString)

d3.dsv(";", "text/plain")

data.forEach(function(e,i)
 data[i].time = time_parse(e.date);
)

)

Ideally I want the data to look like this when logged:

Time, Number1, Number2, Number3, Number4, Number5
00:00, 1.168200, 1.168240, 1.168140, 1.168230, 0
etc

What is the flaw in my thinking and can anyone offer advice on how to solve my problem and similar problems in the future?

Note: I am new to Javascript and d3 and although I have been able to work through most of the documentation involving drawing svgs, creating axis and scales, transitions etc with no problems, I am struggling to get my head around actually getting data from real sources (e.g the internet) and processing them into something workable. Please heavily critique anything I have said and offer advice, I want to learn.

It's not clear what version of d3 you are using, you reference the fetch API, but some of the code you have looks like d3v3 and v4 in the question code (which could be the problem) which doesn't use the fetch API. In any event, I'll go through v5, but also versions 4 and 3.

In all of these your thoughts look pretty close based on the code blocks you have. We need to:

• we read in the dsv as text,


• add headers (with an end of line n),


• and run everything through a dsv format function that will use a ; as a delimiter.

no need for d3.csv.parse though as in your question code block

In all the below I drop the date formatting for simplicity (oops, left it in the v5 demo).

Because of the use of d3-fetch module in d3v5, this approach is a bit different than the more closely related d3v3/v4 (closely related in that they both use the d3-request module, otherwise there's a fair bit of difference).

d3-fetch: d3v5

With d3v5, using the d3-fetch module the process could look like:

var dsv = d3.dsvFormat(";");
var headers = ["time;number1;number2;number3;number4;number5;n"]

d3.text("dsv.dsv").then(function(text) 
 var data = dsv.parse(headers+text);

 console.log(data); 
 console.log(data.columns);

)

Example

d3-request: d3v4

There's a bit more flexibility with d3v4 for this.

If we look at the API docs, we see that d3.csv is equivalent to:

d3.request(url)
 .mimeType("text/csv")
 .response(function(xhr) return d3.csvParse(xhr.responseText, row); );

(docs)

So if we create a new format with d3.dsvFormat we can run the content through the format and get our data, we can also tack on the headers in this process, all in one step:

var dsv = d3.dsvFormat(";");
var headers = ["time;number1;number2;number3;number4;number5;n"]

d3.request("dsv.dsv")
 .mimeType("text/plain")
 .response(function(data) return dsv.parse(headers + data.response) )
 .get(function(data) 
 // use data here: 
 console.log(data); 
 console.log(data.columns);
 ); 

Example

This might be the more atypical approach, so we could emulate the way I did it with v5 above:

var psv = d3.dsvFormat(";");
var headers = ["time;number1;number2;number3;number4;number5;n"]

d3.text("dsv.dsv", function(error,data) 
 var data = psv.parse(headers + data.reponse)

 console.log(data); 
 console.log(data.columns);

)

Example

d3-request: d3v3

As with the second option in d3v4 above and d3v5, we can parse in text and then run it through the dsv format function (here we only need to account for changes in d3 namespace between v3/v4):

var dsv = d3.dsv(";","text/plain");
var headers = ["time;number1;number2;number3;number4;number5;n"]

d3.text("dsv.dsv", function(error,text) 
 var data = dsv.parse(headers+text);
 console.log(data); 
 // console.log(data.columns) // -> no columns property in v3
)

Example

Note

The ; at the end of each row will create an empty column as a value is expected after it before the next row.