json, csv, and extended character sets

For those of you interested in json and csv loading, not all files can be read without first stating their encoding. In cases where the file has a different encoding than the default utf-8, you can expect an error message when the interpreter encounters the first special character. An error like UnicodeDecodeError: 'utf8' codec can't decode byte (...), is mitigated by including the encoding standard this way: open(filename, 'r', encoding='ISO-8859-15').

ISO-8859-15 is commonly used and worth trying first, but is not the only possibility. If you continue to get errors like UnicodeDecodeError then try other types of character encoding, read about them at wikipedia. Also here a pycon talk and direct explanation by Ned Batchelder

This snippet combines the complimentary data from two files. The csv contains group, type and named type data (ie. metal/non-metal and sub type) and combines it with the content of a more elaborate .json.

	import csv
	import json
	some_dict = {}
	filename = "periodictabledump2.csv"
	with open(filename, 'r', newline='') as csvfile:
	ofile = csv.reader(csvfile, delimiter=',')
	ofile.__next__()
	for row in ofile:
	try:
	items = {"group": row[4], "type": row[5], "named_type": row[6]}
	some_dict[row[2]] = items
	except:
	print("failed at")
	print(row)
	filename = "elements.json"
	with open(filename, 'r', encoding='ISO-8859-15') as fp:
	found_json = json.load(fp)
	# for each element in the elements.json update it with the
	# interesting content of the csv per element.
	for element in some_dict.keys():
	found_json[element].update(some_dict[element])
	# reconstruct new json
	with open('elements_joined.json', 'w') as wfile:
	wfile.write(json.dumps(found_json, sort_keys=True, indent=4))

view raw csv_json.py hosted with ❤ by GitHub

Full script with data files here