Requesting data with D3

How to request and parse CSV and JSON data using D3 requests.

Any time a web browser wishes to request a resource, be it an HTML file, a JPEG image or a CSV file, it uses an HTTP request.

For the purposes of this chapter you don't need to understand HTTP requests in detail but if you wish to learn more I recommend Codecademy's guide.

Typically the data (or resource) you wish to request will have a URL (Uniform Resource Locator) such as https://assets.codepen.io/2814973/my-csv.csv.

The URL can also be relative to the index.html of your web application so it might look like data/my-csv.csv

D3 requests

D3 makes requesting data relatively simple. It deals with the HTTP request and can also transform the incoming data into a useful format. For example it can request a CSV (comma separated value) file and transform it into an array of objects.

This chapter will focus on requesting common data formats namely CSV, TSV and JSON. When building data visualisations these are the only formats I use.

Requesting CSV data

CSV files are text files containing tabular data. For example:

rank,workers,company,state_l,city,growth,revenue,industry
1,227,Fuhu,California,El Segundo,158956.9106,195640000,Consumer Products & Services
2,191,Quest Nutrition,California,El Segundo,57347.9246,82640563,Food & Beverage
3,145,Reliant Asset Management,Virginia,Arlington,55460.1646,85076502,Business Products & Services
4,62,Superfish,California,Palo Alto,26042.963,35293000,Software
5,92,Acacia Communications,Massachusetts,Maynard,20690.4578,77652360,Telecommunications
...

You typically export CSV files from spreadsheets and databases. Much of the open data you find on the web is also in CSV format.

Typically (but not always) the first line contains the names of each field (rank, workers, company etc.). The remaining lines contain the data. Each value is separated by a comma.

You can use d3.csv to request a CSV file:

d3.csv('https://assets.codepen.io/2814973/Inc5000+Company+List_2014-top250.csv')
  .then(function(data) {
    console.log(data);
  });

d3.csv accepts the URL as its first parameter and returns a promise object.

A promise object represents an asynchronous operation. An asynchronous operation is an operation whose result is some time in the future. This means that your code can continue to run immediately after initiating the request.

When the browser receives the requested data, the promise is fulfilled and the function that's passed into the promise's .then method is called.

You don't need to fully understand promises to use d3.csv. The main thing to remember is to pass a callback function into the .then method. The callback method is called when the file arrives. Its first parameter is the data.

When the request is fulfilled, D3 converts the incoming CSV file into an array of objects. Each object represents a row of data:

[
  {
    "rank": "1",
    "workers": "227",
    "company": "Fuhu",
    "state_l": "California",
    "city": "El Segundo",
    "growth": "158956.9106",
    "revenue": "195640000",
    "industry": "Consumer Products & Services"
  },
  {
    "rank": "2",
    "workers": "191",
    "company": "Quest Nutrition",
    "state_l": "California",
    "city": "El Segundo",
    "growth": "57347.9246",
    "revenue": "82640563",
    "industry": "Food & Beverage"
  },
  {
    "rank": "3",
    "workers": "145",
    "company": "Reliant Asset Management",
    "state_l": "Virginia",
    "city": "Arlington",
    "growth": "55460.1646",
    "revenue": "85076502",
    "industry": "Business Products & Services"
  },
  ...
]

d3.csv assumes the first row contains field names.

Note that all the numbers are represented by strings. We'll show how to convert these back to numbers later on.

The above array can be joined to HTML or SVG elements. For example:

function update(data) {
  d3.select('#content tbody')
    .selectAll('tr')
    .data(data)
    .join('tr')
    .html(function(d) {
      let html = '<tr>';
      html += '<td>' + d.company + '</td>';
      html += '<td>' + d.industry + '</td>';
      html += '<td>' + d.revenue + '</td>';
      html += '<td>' + d.workers + '</td>';
      html += '</tr>';
      return html;
    });
}

d3.csv('https://assets.codepen.io/2814973/Inc5000+Company+List_2014-top250.csv')
  .then(function(data) {
    update(data);
  });

Row conversion

As pointed out previously, D3 interprets numbers in the CSV file as strings.

You can convert strings to numbers at any point in your code but I recommend doing the conversion straightaway. This can be done by passing a function into d3.csv as the second argument. The function is called on each row of data and you return a new object with any appropriate transformations.

In this example we convert strings to numbers (where appropriate) and rename some of the variable names:

function convertRow(d) {
  return {
    rank: +d.rank,
    workers: +d.workers,
    name: d.company,
    state: d.state_l,
    city: d.city,
    growth: +d.growth,
    revenue: +d.revenue,
    sector: d.industry
  }
}

d3.csv('https://assets.codepen.io/2814973/Inc5000+Company+List_2014-top250.csv', convertRow)
  .then(function(data) {
    console.log(data);
  });

The + operator converts a string to a number.

Now the array of data looks like:

[
  {
    "rank": 1,
    "workers": 227,
    "name": "Fuhu",
    "state": "California",
    "city": "El Segundo",
    "growth": 158956.9106,
    "revenue": 195640000,
    "sector": "Consumer Products & Services"
  },
  {
    "rank": 2,
    "workers": 191,
    "name": "Quest Nutrition",
    "state": "California",
    "city": "El Segundo",
    "growth": 57347.9246,
    "revenue": 82640563,
    "sector": "Food & Beverage"
  },
  {
    "rank": 3,
    "workers": 145,
    "name": "Reliant Asset Management",
    "state": "Virginia",
    "city": "Arlington",
    "growth": 55460.1646,
    "revenue": 85076502,
    "sector": "Business Products & Services"
  },
  ...
]

Requesting TSV data

TSV data is tab separated value data and is treated in a similar manner to CSV. Use d3.tsv to load TSV data.

Requesting JSON data

JSON is a file format that closely mirrors JavaScript arrays and objects. It allows for nested structures which gives it an edge over tabular file formats. JSON is commonly used as a file format by APIs.

Here's an example of a JSON file that's located at https://assets.codepen.io/2814973/my-json.json:

[
  {
    "name": "Paris",
    "indicator1": 4030,
    "indicator2": 13.45
  },
  {
    "name": "Tokyo",
    "indicator1": 3912,
    "indicator2": 45.41
  },
  {
    "name": "New York",
    "indicator1": 19481,
    "indicator2": 32.53
  }
]

You request a JSON file using d3.json:

d3.json('https://assets.codepen.io/2814973/my-json.json')
  .then(function(data) {
    console.log(data);
  });

When the JSON file arrives, D3 converts it into a JavaScript array or object:

[
  {
    "name": "Paris",
    "indicator1": 4030,
    "indicator2": 13.45
  },
  {
    "name": "Tokyo",
    "indicator1": 3912,
    "indicator2": 45.41
  },
  {
    "name": "New York",
    "indicator1": 19481,
    "indicator2": 32.53
  }
]

In this example, the JSON file represents an array, so you can join it to HTML or SVG elements:

function update(data) {
  d3.select('#content tbody')
    .selectAll('tr')
    .data(data)
    .join('tr')
    .html(function(d) {
      let html = '<tr>';
      html += '<td>' + d.name + '</td>';
      html += '<td>' + d.indicator1 + '</td>';
      html += '<td>' + d.indicator2 + '</td>';
      html += '</tr>';
      return html;
    });
}

d3.json('https://assets.codepen.io/2814973/my-json.json')
  .then(function(data) {
    update(data);
  });

Unlike CSV data, JSON data isn't necessarily an array of objects so d3.json doesn't support row conversion functions.

Further notes

If the resource has a different domain (e.g. the https://assets.codepen.io part) to your web application, the server might not be willing to fulfil the request due to CORS restrictions. However in this chapter we request files that are free from CORS restrictions.

Another thing to bear in mind is if you're developing your code locally (rather than on CodePen or similar) you'll need to have a local webserver running. If you're not sure how to do this I recommend reading Fundamentals of HTML, SVG, CSS and JavaScript for Data Visualization.