I watched a really great video about writing an ELF file from scratch, byte by byte, to understand the structure of the file format. It kind of left me wanting a bit more, since I still wasn’t sure about some concepts like the difference between sections and segments. I mean, they’re synonyms! Why did they choose that!? Maybe runtime information vs linktime information? Anyways…

An ELF file is made up of a few parts. There is an ELF header, ELF Program headers, ELF Section headers, then just binary data. The ELF header comes first. It doesn’t matter what comes next, since you specify where program and section header tables are in the ELF header. By table, I just mean a sequence of structures, each of which are the same size. A program header table is just a sequence of struct Elf64_Phdr, for example.

ELF Segments are a runtime notion. They are portions of the ELF file that get mapped from the ELF file to memory when you start up the program. An ELF Segment is simply a contiguous block of the ELF file described by one ELF Program header. The ELF Program headers are used by the kernel to map the portion of the file that the program headers reference to memory at runtime.

An ELF Section is contiguous block of the ELF file that is described by an ELF Section header, used for linkers, debuggers etc. The memory ELF Sections reference in the ELF file can also be part of segments. ELF Sections headers have nothing to do with runtime, in fact, you can strip all the section header information with:

strip —strip-section-headers ./executable

If you run that stripped executable, it will run perfectly.

An ELF Section can be code, data, or information about the code, like symbols used by linkers. Here is an example of writing an object file, byte by byte. It includes the ELF header, section headers, a symbol table, code, a section header string table and a string table. All of that allows gcc to link with a main.c that calls the function described in that object file.

I only really understood the format once I wrote it out myself. I really recommend watching the elf by hand video, then writing an object file in a similar fashion and get gcc to link with it. Doing it in C is really handy, since you can include elf.h and that header comes with all the structs you need. If you don’t know any assembly, I included print function at print_string.s and you can use the following to get the binary form:

as print_string.s -c -o print_string.o
objdump -S print_string.o

This is an example object file layout:

Note that you can’t run this file, because the OS has no idea how to map code and data to memory when the program starts.

Try writing it by hand using elf.h! It can be annoying to get all the sections right if you do do it by hand, so good luck.

Programs like Valgrind often use a technique called shadow memory that “shadows” every byte of memory. It gives us some useful metadata about the byte that we are accessing, like whether or not it has been allocated. During execution these tools will update the shadow memory when a memory read, write, free or allocation occurs.

A lot of tools use this technique for catching different issues like data races and security related issues. The problem with Valgrind is that it’s painfully slow from all the virtualization (~20x slowdown). From reading up on Valgrind, it takes in your binary, converts it into an internal representation and runs it with a bunch of instrumentation to analyze memory accesses among other things. Valgrinds main benefit is that you don’t need the source code, just the binary. It’s also very thorough. From what I’ve read it can also do some profiling.

I have the source code, and due to working on machine learning workloads with big tensors doing matrix multiplication and convolutions, I need speed. Instead of using Valgrind, I learned how to use address sanitizer instead. Address Sanitizer (ASan) is compile-time instrumentation, and is a lot more lightweight. You need the source code and it needs to work in your compiler. GCC supports ASan. It only has a ~2x slowdown. After compiling with -fsanitize=address I quickly got useful information.

From this I assumed I was allocating to little memory for what I needed.

ERROR: AddressSanitizer: heap-buffer-overflow ...
... WRITE of size 692224 ...
... mlpack/methods/ann/dag_network_impl.hpp:593
... allocated by thread T0 here:
... mlpack/methods/ann/dag_network_impl.hpp:509

I think this is telling me at line 593 I’m trying to write beyond some memory allocation that I allocated at line 509.

509: layerOutputMatrix = MatType(1, batchSize * forwardMemSize);

This gives me the hint that forwardMemSize is probably the wrong. It turns out when I’m computing the size of a concatenation operation, I was computing it like so:

size_t concatSize = layers[i]->InputDimensions()[0]
for (size_t j = 1; j < layers.size(); j++)
   concatSize += layers[i]->InputDimensions()[j];

when it should clearly be done by multiplication. In retrospect this is a very stupid bug that took way to long to find, but after learning about how address sanitizer works and how to use it saved me a lot of time. The lesson here is to get better a using debugging tools.

Useful links

• Useful wiki on ASan

• Table comparing ASan to tools like Valgrind

• Blog on why you should use ASan

Gshare uses two structures to make predictions. A pattern history table and a global history register.

Pattern History Table

A pattern history table (PHT) is an array of 2-bit saturating counters. Saturating arithmetic is when the output of any operation has a fixed minimum and maximum value. With a 2-bit saturating counter, this means that there is no overflow. If the value of the counter is 11 and we increase the value, it is still 11. We use this saturating arithmetic to represent a finite state machine that represents four states: strongly taken, weakly taken, weakly not-taken and strongly not-taken.

Here is the finite state machine of a 2-bit saturating counter:

Every time the output of a branch instruction is evaluated, the PHT gets updated. If it’s current state is weakly taken, and that branch was evaluated and was actually taken, we update the counter to be in the strongly taken state.

This structure keeps track of a branches local history. If it’s state is strongly not-taken, then it is very likely that that branch will not be taken. We can use this structure to make predictions on a given branch. We could index into this array of counters with the program counter (PC) and make a prediction based on this local history.

Global History Register

It has been shown that taking global history and local history into account can get you better predictions. Global branch history is a typically sequence of 1’s and 0’s, where each bit represents whether the ith last branch was taken. Instead of looking at what the current branch has done so far, like a PHT, global history looks at what the last N branches has done.

We can represent this as a shift register. Every time a branch executes, we shift the bits to the left, and add the branch outcome to the end.

Instead of indexing with program counter, gshare XORs the global history with the program counter, and uses this output to index into the PHT.

Here is the full diagram:

Regular Convolution

A normal convolution is like a linear layer inside a neural network, except that it works on patches of the input instead of the taking the entire image into account, like a fully connected layer. The weights in a convolutional layer is made up of kernels. At the start of training these kernels are made up of random numbers, but after training allow the network to identify patterns that make it useful for image classification or object detection.

There are m kernels during a convolution. where m is the number of output channels. Each kernel has a width and height (like 3x3 or 5x5) and a number of channels, which is the same number of input channels. For example, with a 224x224 image with 3 channels (rgb), we could ‘convolve’ the image with 8 3x3 kernels, outputting a 222x222x8 feature map (in this case there is no padding, no dilation and the stride is just 1). A convolution multplies each number in it’s kernel by it’s corresponding place in the image, summing those multplications up, and then shifting the kernel over by one (or whatever the stride is), and repeating for the entire input.

The most common convolution you’ll see is the 3x3 convolution (VGG, Mobilenet, YOLO as examples). It reduces the amount of multiplications needed, with the same number of input/output channels. One 5x5 convolution will take in the same amount of the image as two 3x3 kernels done one after the other, but doing the two 3x3 convoltuions instead results in less multiplications. For example, in a 224x224x3 image, doing a 5x5 convolution with 16 output channels results in 220*220*3*5*5*16 = 580,800,000 multiplications, while two 3x3 convolutions with the same number of output channels results in (222*222*3*3*3*16)+(220*220*3*3*3*16) = 42,199,488 multiplications. The second layer using 3x3 convolutions however will effectively be able to see a 5x5 part of the oringal feature map.

I’ll use 3x3 convolutions as the example for the rest of the post, with no padding/dilation and the strides will be set to 1.

Groups

Groups are another argument on how to describe a convolution operation. You’ll see it in the PyTorch documentation for Conv2d. Having n groups means splitting the input channels into n pieces, and using n groups of kernels, where each group of kernels will only process its corresponding input channel group.

Depthwise Separable Convolution

It turns out we can reduce the amount of computation required in a convolution using depthwise separable convolutions, which takes groups to the extreme shown in the MobileNet paper.

I’ll use a regular convolution to illustrate why this reduces computational cost. If we have an input feature map of shape 28x28x8, and a convolutional layer of 3x3 convolutions with 16 output channels, we have to do 26*26*3*3*8*16 = 778,752 multiplications. We will get a 26x26x16 feature map as the output.

There are two steps to a depthwise separable convolution. First, we perform a depthwise convolution, and then a pointwise convolution, which will get us the same complexity in our neural net but reduce the number of calculations needed.

We seperate the incoming feature map into n groups, where n is the number of input channels. This means that each group will only have one channel. We can apply one 3x3 convolution to each group. This is the depthwise convolution. If we perform this for the same example, we will output a 26x26x8 feature map.

Next we will use 16 1x1 convolution kernels, which will be our pointwise convolution. This will output a 26x26x16 feature map, with exactly the same shape as doing a regular 3x3 convolution, each of the 16 kernels having 8 channels.

How does this improve performance? The depthwise convolution uses 26*26*3*3*8 = 48672 multiplications. The pointwise layer uses 26*26*8*16 = 86528 multiplications. In total 135200 multiplications, which is an 82% reduction in mulplications. The MobileNet paper shows that the reduction will be

where N is the number of kernels and D_K is the size of the kernels. Since most layers use 3x3 kernels, as you increase the number of output channels you approach 89% reduction in multiplications.

Conclusion

Using depthwise separable 3x3 convolutions massively reduces the amount of computation needed to perform these operations, while maintaining the same complexity in the neural network.

Why?

Cyclic redundancy checks are numbers you append to moving data (either on a network, or even transferring from storage to memory) that allows you to confirm that the data has no errors. Moving data around without some sort of error detection would be pretty useless.

How?

The maths involved is interesting but because I’m not an expert on finite fields I’m not going to delve too deeply into things. You can find decent books on the theory behind Galois fields and proofs on how well CRC works on the internet. The main principle of cyclic redundancy checks is to represent that data as a polynomial in GF(2). This means that the data will be a polynomial, where the coefficients are either 1 or 0. Since data is represented in binary in computers, this is perfect.

Here is an example:

We then decide on a generator polynomial. We’ll use this to calculate the number that we append at the end of the message. An example polynomial might be:

We will use this polynomial to divide into our data polynomial. This is the exact same long division algorithm you would have learned in school for dividing polynomial, and finding their remainder. We’re using GF(2) arithmetic. Subtraction and addition are equivalent and are the same as an XOR operation.

The remainder 01 is our redundant information that we will append to our outgoing message 11010. The sender will transmit 1101001 and the receiver, who also has the generator polynomial will divide 111 into 11010 and will check to see if the remainder is the same as what was received. I left out the quotient because it’s irrelevant to CRC. All we need is the remainder.

Basic Implementation

Our generator polynomial for CRC32 will be 32bits long as the name implies. The polynomial we’ll be using is 0x04C11DB7. The biggest thing to note here is that in an actual implementation, we don’t need the leading 1, since everytime we XOR 1 with 1 it will be 0.

# Basic implementation if crc32
def crc32(message:bytearray, poly:int):
    bitmask = 0xFFFFFFFF
    crc = 0

    for byte in message:
        for _ in range(8):
            b = byte & (1<<7) != 0
            divide = bitmask if (crc & (1<<31) != 0) else 0
            crc = (crc << 1) | b
            crc ^= (poly & divide)
            byte <<= 1
    return (crc & bitmask) # to keep it at 32bits long

Some little details

There are a few things that can be improved upon above, before we go implementing anything.

First, it’s going to be a pain having to wait for the entire message to come in before we get to calculate the remainder. We want this to be really fast, and start calculating the remainder while the message is coming in.

To do this, in our little example above, we will append 2 zeroes to our outgoing message, before we calculate the remainder. Like so

So, our outgoing message will be 1101011. This allows will allow the receiver to calculate the remainder as the message is coming in. If the remainder is 0, we know (or most likely) have the correct data.

What happens if leading 0’s get added/deleted during transmission? It will still be divisible by the generator, but obviously the wrong data. The solution is to prepend a certain amount of 1’s (in our case 32, since we’re going to implement a 32 bit CRC) to the data.

What happens if ending 0’s get added/deleted during transmission? Well this wouldn’t be a problem if our message (data + remainder) ended with a 1, but if it ends with a 0 this error may go undetected. In this case, we will end the function with an XOR of some value to fix this problem.

The resulting code becomes:

# Better implementation of crc32, leading/trailing zeros
def crc32_improved(message:bytearray, poly:int, init:int=0, final_xor:int=0):

    bitmask = 0xFFFFFFFF
    crc = init

    for byte in message:
        for _ in range(8):
            b = bitmask if byte & (1<<7) != 0 else 0
            divide = bitmask if (crc & (1<<31)) != 0 else 0
            crc = (crc << 1) ^ (poly & (b ^ divide))
            byte <<= 1
    return (crc & bitmask) ^ final_xor

There are some other optimizations you could make. You could make a look up table that does some caching like so:

from functools import lru_cache
@lru_cache
def create_lut(poly): return [crc32_improved(bytearray([x]), poly) for x in range(256)]
# Implementation of crc32 with look up table
def crc32_lut(message:bytearray, poly:int):
    """
    Generates a crc with a look up table for improved speed
    """
    bitmask = 0xFFFFFFFF
    crc = 0

    lut = create_lut(poly)
    for m in message:
        index = (int(m) ^ (crc >> 24)) & 0xFF
        crc = (crc << 8) ^ lut[index]

    return crc & bitmask