opencv - Kinect + Python - Fill depth for shadows -

the kinect camera returns depth image whole view. due way image captured, small areas invisible camera. areas 2047 returned.

i want fill areas value left of them - value area. have depth numpy uint16 array. trivial solution be:

for x in xrange(depth.shape[1]):   y in xrange(depth.shape[0]):     if depth[y,x] == 2047 , x > 0:       depth[y,x] = depth[y,x-1] 

this takes around 16 seconds execute (raspberry 2) per 640 x 480 frame.

i came solution using indexes:

w = numpy.where(depth == 2047) w = zip(w[0], w[1]) index in w:   if depth[index] == 2047 , index[1] > 0:     depth[index] = depth[index[0],index[1] - 1] 

this takes around 0.6 seconds execute test frame. faster still far perfect. index computation , zip take 0.04 seconds, main performance killer loop.

i reduced 0.3 seconds using item():

for index in w:   if depth.item(index) == 2047 , index[1] > 0:     depth.itemset(index, depth.item(index[0],index[1] - 1)) 

can improved further using python (+numpy/opencv)? compared how fast simple filtering is, should possible faster 0.05s

you have islands going behind places elements in input array 2. job here fill shadows element right before starting shadows. so, 1 way find out start , stop places of islands , put x , -x @ places respectively, x element right before starting of each island. then, cumsum along rows, fill shodow-islands x. that's there vectorized solution! here's implementation -

# mask of places updated mask = np.zeros(np.array(depth.shape) + [0,1],dtype=bool) mask[:,1:-1] = depth[:,1:] == 2047  # differentiation along second axis , island start , stops diffs = np.diff(mask.astype(int),axis=1) start_mask = diffs == 1 stop_mask = diffs == -1  # mapping array has island places filled start-1 element map_arr = np.zeros_like(diffs) map_arr[start_mask] = depth[start_mask] map_arr[stop_mask] = -depth[start_mask] map_filled_arr = map_arr.cumsum(1)[:,:-1]  # use mask created earlier selectively set elements map array valid_mask = mask[:,1:-1] depth[:,1:][valid_mask] = map_filled_arr[valid_mask] 

benchmarking

define functions :

def fill_depth_original(depth):     x in xrange(depth.shape[1]):         y in xrange(depth.shape[0]):             if depth[y,x] == 2047 , x > 0:                 depth[y,x] = depth[y,x-1]  def fill_depth_original_v2(depth):     w = np.where(depth == 2047)     w = zip(w[0], w[1])     index in w:       if depth[index] == 2047 , index[1] > 0:         depth[index] = depth[index[0],index[1] - 1]  def fill_depth_vectorized(depth):      mask = np.zeros(np.array(depth.shape) + [0,1],dtype=bool)     mask[:,1:-1] = depth[:,1:] == 2047      diffs = np.diff(mask.astype(int),axis=1)     start_mask = diffs == 1     stop_mask = diffs == -1      map_arr = np.zeros_like(diffs)     map_arr[start_mask] = depth[start_mask]     map_arr[stop_mask] = -depth[start_mask]     map_filled_arr = map_arr.cumsum(1)[:,:-1]      valid_mask = mask[:,1:-1]     depth[:,1:][valid_mask] = map_filled_arr[valid_mask] 

runtime tests , verify outputs :

in [303]: # create random array , copy profiling vectorized method      ...: depth = np.random.randint(2047-150,2047+150,(500,500))      ...: depthc1 = depth.copy()      ...: depthc2 = depth.copy()      ...:   in [304]: fill_depth_original(depth)      ...: fill_depth_original_v2(depthc1)      ...: fill_depth_vectorized(depthc2)      ...:   in [305]: np.allclose(depth,depthc1) out[305]: true  in [306]: np.allclose(depth,depthc2) out[306]: true  in [307]: # create random array , copy profiling vectorized method      ...: depth = np.random.randint(2047-150,2047+150,(500,500))      ...: depthc1 = depth.copy()      ...: depthc2 = depth.copy()      ...:   in [308]: %timeit fill_depth_original(depth)      ...: %timeit fill_depth_original_v2(depthc1)      ...: %timeit fill_depth_vectorized(depthc2)      ...:  10 loops, best of 3: 89.6 ms per loop 1000 loops, best of 3: 1.47 ms per loop 100 loops, best of 3: 10.3 ms per loop 

so, second approach listed in question still looks winning!