Sparse Initialization example with Simple Python

We’ll use a 3-layer neural network, but initialize it with sparse weights.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import random
 
# Simulating sparse weight initialization
def sparse_initialize(rows, cols, sparsity=0.8):
    weights = []
    for i in range(rows):
        row = []
        for j in range(cols):
            if random.random() < sparsity:
                row.append(0.0)  # Mostly zeros
            else:
                row.append(random.uniform(-0.1, 0.1))  # Small random values
        weights.append(row)
    return weights
 
# Example: 4 input neurons, 5 hidden neurons
input_to_hidden = sparse_initialize(5, 4, sparsity=0.7)
 
# Display
print("Sparse Initialized Weights (Hidden Layer):")
for row in input_to_hidden:
    print(row)

Output:

Sparse Initialized Weights (Hidden Layer):
[0.0, 0.0, 0.03255273994082253, 0.048024808700993404]
[0.0, 0.0, 0.0, 0.0]
[0.0, 0.0, 0.09314012611398312, 0.0]
[-0.06676975747645872, -0.04194356851680425, 0.0, -0.023752588882273717]
[0.0, 0.0, 0.0, 0.0]

Sparse Initialization applicability in Neural Network – Basic Math Concepts