Make select Faster

[mmontagna]

What

Use Psycopg2 directly for PG selects in order to improve performance.

Why

sqlalchemy ads ~1ms of additional latency for basic selects and more for queries which return large result sets.

Without sqlalchemy

➜  test git:(marco-make-select-faster) ✗ lore python sql_test.py
...
raw rows average runtime 0.23196000000000003 ms

With sqlalchemy

➜  test git:(marco-make-select-faster) ✗ lore python sql_test.py
...
raw rows average runtime 1.371959 ms

sql_test.py

import lore, lore.io
import time
import numpy as np
runtimes = []

main = lore.io.main


for item_id in range(0, 10000):
    st = time.time()
    results = main.select(sql='select 1 where 1 = 1;')
    runtime = round((time.time() - st)*1000.0, 2)
    runtimes.append(runtime)

avg_runtime = np.mean(runtimes)

print("raw rows average runtime {} ms".format(avg_runtime))

[ganesh-krishnan]

Wonder if there is value in falling back to SQL alchemy as a final try before giving up completely.

[ganesh-krishnan]

Not a blocker though. This looks good!

[ilystsov]

If self._use_psycopg2 is False, we immediately return the result for the alternative execution.
Merged the two with statements into one line.
if the error isn't about "no results to fetch", the original exception will be automatically re-raised without needing to explicitly call raise e.

[ilystsov]

Suggested change

	def _connection_execute(self, sql, bindings):
	if self._use_psycopg2:
	with self._connection.engine.raw_connection().connection as conn:
	with conn.cursor() as cursor:
	cursor.execute(sql, bindings)
	try:
	return ResultWrapper(cursor.fetchall())
	except psycopg2.ProgrammingError as e:
	if 'no results to fetch' in str(e):
	return None
	raise e
	else:
	return self._connection.execute(sql, bindings)
	def _connection_execute(self, sql, bindings):
	if not self._use_psycopg2:
	return self._connection.execute(sql, bindings)
	try:
	with self._connection.engine.raw_connection().connection as conn, conn.cursor() as cursor:
	cursor.execute(sql, bindings)
	return ResultWrapper(cursor.fetchall())
	except psycopg2.ProgrammingError as e:
	if 'no results to fetch' not in str(e):
	raise