python 3.x - tensorflow: creating variables in fn of tf.map_fn returns value error -

i have questions regarding variable initialization in map_fn.

i trying apply highway layers separately on each individual element in tensor, figure map_fn might best way it.

segment_list = tf.reshape(raw_segment_embedding,[batch_size*seqlen,embed_dim]) segment_embedding = tf.map_fn(lambda x: stack_highways(x, hparams), segment_list) 

now problem fn, i.e. stack_highways, create variables, , reason tensorflow fails initialize variables , give error.

w = tf.variable(tf.truncated_normal(w_shape, stddev=0.1), name='weight')  valueerror: initializer variable body/model/parallel_0/body/map/while/highway_layer0/weight/ inside control-flow construct, such loop or conditional. when creating variable inside loop or conditional, use lambda initializer.  

i pretty clueless now, based on error suppose not scope have no idea how use lambda initializer (i dont know mean). below implementation of stack_highways, advice appreciated..

def weight_bias(w_shape, b_shape, bias_init=0.1):   """fully connected highway layer adopted       https://github.com/fomorians/highway-fcn/blob/master/main.py   """   w = tf.variable(tf.truncated_normal(w_shape, stddev=0.1), name='weight')   b = tf.variable(tf.constant(bias_init, shape=b_shape), name='bias')   return w, b     def highway_layer(x, size, activation, carry_bias=-1.0):   """fully connected highway layer adopted       https://github.com/fomorians/highway-fcn/blob/master/main.py   """   w, b = weight_bias([size, size], [size])   tf.name_scope('transform_gate'):     w_t, b_t = weight_bias([size, size], bias_init=carry_bias)       h = activation(tf.matmul(x, w) + b, name='activation')     t = tf.sigmoid(tf.matmul(x, w_t) + b_t, name='transform_gate')     c = tf.sub(1.0, t, name="carry_gate")       y = tf.add(tf.mul(h, t), tf.mul(x, c), name='y') # y = (h * t) + (x * c)     return y     def stack_highways(x, hparams):   """create highway networks, not create   padding layer in bottom , top,    layers of highways.     args:     x: raw_segment_embedding     hparams: run hyperparameters     returns:     y: segment_embedding   """   highway_size = hparams.highway_size   activation = hparams.highway_activation #tf.nn.relu   carry_bias_init = hparams.highway_carry_bias   prev_y = none   y = none   in range(highway_size):     tf.name_scope("highway_layer{}".format(i)) scope:       if == 0: # first, input layer         prev_y = highway_layer(x, highway_size, activation, carry_bias=carry_bias_init)       elif == highways - 1: # last, output layer         y = highway_layer(prev_y, highway_size, activation, carry_bias=carry_bias_init)       else: # hidden layers         prev_y = highway_layer(prev_y, highway_size, activation, carry_bias=carry_bias_init)   return y 

warmest regards, colman

tensorflow provides 2 main ways of initializing variables:

1. "lambda" initializers: callables return value of initialization. tf provides many nicely packaged ones.
2. initialization tensor values: using currently.

the error message stating need use first type of initializer when using variables within while_loop (which map_fn calls internally). (in general lambda initializers seem more robust me.)

additionally in past, tf.get_variable seems preferred on tf.variable when used within control flow.

so, suspect can resolve issue fixing weight_bias function this:

def weight_bias(w_shape, b_shape, bias_init=0.1):   """fully connected highway layer adopted       https://github.com/fomorians/highway-fcn/blob/master/main.py   """   w = tf.get_variable("weight", shape=w_shape,           initializer=tf.truncated_normal_initializer(stddev=0.1))   b = tf.get_variable("bias", shape=b_shape,           initializer=tf.constant_inititializer(bias_init))   return w, b 

hope helps!