python - numba @njit to update a big dict -
i try use numba function need search on big (10e6) dict (int, int) tuple key.
import numpy np numba import njit myarray = np.array([[0, 0], # 0, 1 [0, 1], [1, 1], # 1, 2 [1, 2], # 1, 3 [2, 2], [1, 3]] ) # lot of shape~(10e6, 2) dict_with_tuples_key = {(0, 1): 1, (3, 7): 1} # ~10e6 keys
a simplified version this
# @njit def update_dict(dict_with_tuples_key, myarray): line in myarray: i, j = line if (i, j) in dict_with_tuples_key: dict_with_tuples_key[(i, j)] += 1 else: dict_with_tuples_key[(i, j)] = 1 return dict_with_tuples_key new_dict = update_dict(dict_with_tuples_key, myarray) print new_dict new_dict = update_dict2(dict_with_tuples_key, myarray) # print new_dict # {(0, 1): 2, # +1 in dict_with_tuples_key # (0, 0): 1, # diag # (1, 1): 1, # diag # (2, 2): 1, # diag # (1, 2): 1, # new myarray # (1, 3): 1, # new myarray # (3, 7): 1 }
it appear @njit not accept dict function arg ?
i'm wondering how rewrite this, specially if (i, j) in dict_with_tuples_key
part search.
njit
means function compiled in nopython
mode. dict
, list
, tuple
python objects , therefore not supported. not arguments , not inside function.
if dict keys different consider using 2d numpy array first axis represents first index of dict-key-tuple , second axis second index. rewrite as:
from numba import njit import numpy np @njit def update_array(array, myarray): elements = myarray.shape[0] in range(elements): array[myarray[i][0]][myarray[i][1]] += 1 return array myarray = np.array([[0, 0], [0, 1], [1, 1], [1, 2], [2, 2], [1, 3]]) # calculate size of numpy array replaces dict: lens = np.max(myarray, axis=0) # maximum values array = np.zeros((lens[0]+1, lens[1]+1)) # create empty array hold indexes in myarray update_array(array, myarray)
since indexed dictionary tuples transition problems indexing array not great.
Comments
Post a Comment