Issue139/monetdbe append #167
Conversation
This is the precision that monetdbe offers. For us it is a shortcoming in pandas that it does not offer other units of precision outside of nanoseconds.
| # cffi_objects assists to keep all in-memory native data structure alive during the execution of this call | ||
| cffi_objects = [] | ||
| cffi_objects = list() |
There was a problem hiding this comment.
i think in an other PR or a previous commit i merged work_objs and cffi_objects, since afaik they do the same thing, or not?
| @@ -48,34 +48,33 @@ class MonetdbTypeInfo(NamedTuple): | |||
| numpy_type: np.dtype | |||
| c_string_type: str | |||
| py_converter: Optional[Callable] | |||
| null_value: Optional[Union[int, np.floating]] | |||
There was a problem hiding this comment.
if you removed that to fix mypy complaining, numpy added new type annotations, and in master i fixed the type annotation for the null field (an optional np.floatable I believe)
| np_col: np.ndarray = np.array([extract(rcol, r) for r in range(result.nrows)]) | ||
| values = [extract(rcol, r) for r in range(result.nrows)] | ||
| np_col: np.ndarray = np.array(values) | ||
| np_mask = np.array([v is None for v in values]) |
There was a problem hiding this comment.
To me, this seems almost the same except for an extra list comprehension.
mask = np_col == type_info.null_value is evaluated in numpy land and should be faster
my guess is you did this to get rid of the monetdbType null value field, see comment above.
There was a problem hiding this comment.
added an issue for this #168
|
|
||
| work_column.type = type_info.c_type | ||
| work_column.count = column_values.shape[0] | ||
| work_column.name = ffi.new('char[]', column_name.encode()) | ||
| if type_info.numpy_type.kind == 'U': | ||
| if type_info.numpy_type.kind == 'M': | ||
| t = ffi.new('monetdbe_data_timestamp[]', work_column.count) |
There was a problem hiding this comment.
just a comment, no action required, but this function is getting a bit messy, we should split it up in the future.
| np_col: np.ndarray = np.array([extract(rcol, r) for r in range(result.nrows)]) | ||
| values = [extract(rcol, r) for r in range(result.nrows)] | ||
| np_col: np.ndarray = np.array(values) | ||
| np_mask = np.array([v is None for v in values]) |
There was a problem hiding this comment.
added an issue for this #168
pull request for new numpy/type features