Simple csv file can't be parsed correctly because of Sniffer() parameters

[lap1nou]

Greetings,

First of all, thank you for the work you are doing on this library.

I have the following CSV file, where the delimiter is a colon : :

test:test2
value1:value2

I'm now using the tablib library to import it:

tmp = open('test.csv', 'r')
tmp2 = tablib.Dataset().load(tmp, format='csv')

But the file is wrongly parsed, because the csv library did not sniffed the delimiter correctly:

test:test2   
-------------
value1:value2

I saw that you are passing 2048 bytes of samples to be sniffed (

tablib/src/tablib/formats/_csv.py

Line 58 in 4bbb35a

csv.Sniffer().sniff(stream.read(2048), delimiters=delimiter or cls.DEFAULT_DELIMITER)

), I now tested to manually sniff the CSV using only the first line:

dialect = csv.Sniffer().sniff(tmp.readline())

The delimiter is now indeed recognized as a colon :.

My question is: shouldn't you pass the first line to sniff only (using readline()) ? Or maybe fallback on the actual behaviour only if the headers is not present in the file ?

Related issue: #502

Regards

[claudep]

I think we are open to try better heuristics to detect delimiters, as long as current tests still pass, of course.

[lap1nou]

@claudep After analysis I think I found a bug.

There:

tablib/src/tablib/formats/_csv.py

Line 41 in 8dcb87f

kwargs.setdefault('delimiter', cls.DEFAULT_DELIMITER)

, you are setting a default value if the delimiter parameter is not set, however if someone decide to pass a dialect in the kwargs, the delimiter parameter won't be present, and so the default value you are setting will take precedence over the dialect passed by the user, here is an example using the following CSV:

test:test2
value1:value2

dialect = csv.Sniffer().sniff(csv_file.readline())
csv_file.seek(0)

data = tablib.Dataset().load(
                csv_file, format='csv', dialect=dialect
)

Since no delimiter are passed, tablib will choose the deault one ,, but we actually have the delimiter : in the dialect, so the default delimiter will take precedence over the dialect one and it will fail to correctly import the CSV.

Do you want me to submit a PR to fix this bug ?

[claudep]

Absolutely, if you can show the bug with a regression test, any PR is welcome!