Google CTF 2018 Proprietary Format write-up

Introduction

When we peek at file given by the challenge, the file starts with GCTF(which might mean google ctf), and we don't find any noticable information from this file. So for more information, we tried to send some random data to remote server.

a
Expected P6 as the header

The server expects us to send P6 as the header. So we tried to send P6 and some random data.

P6
a
Expected width and height

The server expects us to send width and height, so we sent it.

P6
300 300
a
Expected 255

Now the server expects 255, so we sent 255 instead of a. Then the server starts to receive data. So we tried to limit the width and height to 1 and send some random data.

[+] Opening connection to proprietary.ctfcompetition.com on port 1337: Done
[DEBUG] Sent 0x3 bytes:
    'P6\n'
[DEBUG] Sent 0x4 bytes:
    '1 1\n'
[DEBUG] Sent 0x4 bytes:
    '255\n'
[DEBUG] Sent 0x3 bytes:
    '\x11' * 0x3
[DEBUG] Received 0x10 bytes:
    00000000  47 43 54 46  01 00 00 00  01 00 00 00  00 11 11 11  │GCTF│····│····│····│
    00000010
[*] Closed connection to proprietary.ctfcompetition.com port 1337

It receives 3 bytes and prints out data.

Observation

We tried to figure out what the result that the server gives means. By testing several test data, we figured out some information.

1. The server receives $3 \times width \times height$ length data.

2. The result contains GCTF at start, 4 bytes which indicate width, 4 bytes which indicate height, and the result data after 12 bytes.

   So we guessed that the server receives 24-bit color mapped picture, compress that data, and prints out GCTF+width+height, and result.

Analysis

We tried giving the server random data of width 2, height 2. Then, we figured out some rules of how the server compresses our data.

Input : ffffffffffffffffffaaaaaa
Output : 
47435446
02000000
02000000
08ffffff
00aaaaaa

Input : ffffffaaaaaaffffffffffff
Output :
47435446
02000000
02000000
02ffffff
00aaaaaa

Input : ffffffaaaaaaffffffaaaaaa
Output : 
47435446
02000000
02000000
0affffff
00aaaaaa
00aaaaaa

Input : aaaaaaffffffaaaaaaffffff
Output : 
47435446
02000000
02000000
0aaaaaaa
00ffffff
00ffffff

Input : aaaaaaffffffffffffffffff
Output : 
47435446
02000000
02000000
01ffffff
00aaaaaa

Input : ffffffffffffaaaaaaffffff
Output : 
47435446
02000000
02000000
04ffffff
00aaaaaa

Input : ffffffffffffaaaaaaaaaaaa
Output : 
47435446
02000000
02000000
0cffffff
00aaaaaa
00aaaaaa

The input and output are expressed in hex values, and output is split with newline after every 4 bytes. We figured out the result follows the rules:

1. 4 bytes are indexed with 1, 2, 4, 8 in order.
2. Choose dominant color. If the number of dominant color is same with the number of second dominant color, choose the color which appears faster.
3. Sum up indexes of non-dominant colors. For example, if input is ffffff/ffffff/aaaaaa/aaaaaa, then we get dominant color ffffff, the sum of indexes is 4+8=0xc.
4. Concatenate the summation we calculated above next to the dominant color. Then we reverse the bytes and print out to result.
5. We now concatenate a null byte next to non-dominant color. Then we reverse the bytes and print out to result. We repeat this task sequentially for every non-dominant color.

We also tried other data and we figured out if two color data is close enough(up to about 0x12 difference in color bytes), then they are considered as same color.

After we figured out the compression rules of 2 x 2 images, we started to test 4 x 4 images. Then, we figured out the following rules.

1. 4 pieces(which are 2 x 2 images) are indexed with 1, 2, 4, 8 in order.
2. Check these pieces if one piece is filled with a single color. If there are no such piece, print 0x0f and follow the rules of 2x2 in order.
3. If at least one piece is filled with a single color, sum up indexes of pieces that are not consisted of a single color. Then, we print the reverse of color+sum, and we follow the rules of 2x2 in order, excluding pieces of single color.

Next we expanded our test data to 8 x 8, and we figured out that the compression algorithm is in recurrence relation(with base condition with 2 x 2 piece).

Rules

Basic condition

1. If two colors have difference lesser than 0x12 in their bytes, they are considered as same color.

Rules in 2 x 2

1. 4 colors are indexed with 1, 2, 4, 8 in order
2. Choose one dominant color. Next, we write that color in front, with their bytes reversed.
   ex) If the dominant color is 0x343536:
   If the dominant color is at 2, 4, then the compression result is 0x09363534. (0x09 = 1+8)

Rules in 4 x 4

1. 4 pieces(2 x 2 images) are indexed with 1, 2, 4, 8 in order.

2. We inspect 4 pieces and check if there is a piece filled with a single color. If such piece doesn't exist, add 0x0f to result.

3. If there are at least one piece filled with a single color, sum up numbers of indexes of pieces that are not filled with single color. For example, if 2, 4 pieces are not filled with a single color, the result is 0x09.

   ex) 0x11 0x11 0x33 0x33

     0x11  0x11  0x33  0x55
     0x55  0x55  0x77  0x77
     0x99  0x99  0xbb  0xbb

   (I wrote one byte assuming 3 bytes of color are all same. (0x11 = 0x111111))
   1) If there is a piece filled with a single color : at index 1 :arrow_right: 0x0e111111 (If there is no such piece, just add 0x0f)
   2) We checked the first piece is filled with a single color : pass.
   3) dominant : 0x33 : 0x08333333

    non dominant : 0x00555555

   4) dominant : 0x55 : 0x0c555555

    non dominant : 0x00999999
    non dominant : 0x00999999

   5) dominant : 0x77 : 0x0c777777

    non dominant : 0x00bbbbbb
    non dominant : 0x00bbbbbb

Rules in n x n

1. If n is not 2^k, we fill empty spaces with 0x000000 padding.
2. We now split the data with 2 x 2, follow the recurrence relation defined in 4 x 4 rules, with base condition rules defined in 2 x 2 rules.

Solve

We now know how the compression algorithm works. Now we write decompress code and get the flag.

from pwn import *

data = open("flag.ctf", "rb").read()

seek = 12

# color_data[height][width]
color_data = [[0x00 for i in xrange(1024)] for j in xrange(1024)]

# fill color from sx(width), sy(height) to tx, ty
def fill_color(sx, sy, tx, ty, color):
        for j in xrange(sy, ty):
            color_data[j][i] = color

# do recursion from sx(width), sy(height) with size(current block size)
def do_recursion(sx, sy, size):
    global data
    global seek
    global color_data
    # base condition
    if size == 2:
        byte_filter = bin(ord(data[seek]))[2:]
        byte_filter = "0"*(4-len(byte_filter)) + byte_filter
        seek += 1
        byte_color = u32(data[seek:seek+3]+"\x00")
        seek += 3
        for i in xrange(4):
            if byte_filter[3-i] == '1':
                color_data[sy+i/2][sx+(i%2)] = u32(data[seek+1:seek+4]+"\x00")
                seek += 4
            else:
                color_data[sy+i/2][sx+(i%2)] = byte_color

    # recurrence relation
    else:
        byte_filter = bin(ord(data[seek]))[2:]
        byte_filter = "0"*(4-len(byte_filter)) + byte_filter
        seek += 1
        if byte_filter == "1111":
            do_recursion(sx, sy, size/2)
            do_recursion(sx+size/2, sy, size/2)
            do_recursion(sx, sy+size/2, size/2)
            do_recursion(sx+size/2, sy+size/2, size/2)
        else:
            byte_color = u32(data[seek:seek+3]+"\x00")
            seek += 3
            for i in xrange(4):
                if byte_filter[3-i] == '1':
                    do_recursion(sx+(i%2)*size/2, sy+(i/2)*size/2, size/2)
                else:
                    fill_color(sx+(i%2)*size/2, sy+(i/2)*size/2, sx+(i%2)*size/2+size/2, sy+(i/2)*size/2+size/2, byte_color)

do_recursion(0, 0, 1024)

bmp_header = "42 4D B6 FC 0A 00 00 00 00 00 36 00 00 00 28 00 00 00 58 02 00 00 90 01 00 00 01 00 18 00 00 00 00 00 80 FC 0A 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00".replace(" ", "")
bmp_header = bmp_header.decode("hex")

# result is color data
with open("result.bmp", "w") as f:
    f.write(bmp_header)
    for i in xrange(400):
        for j in xrange(600):
            dat = hex(color_data[400-i][j])[2:]
            if len(dat) % 2 == 1:
                dat = "0"+dat
            dat = dat.decode("hex")
            assert len(dat) <= 3
            dat = "\x00"*(3-len(dat))+dat
            f.write(dat)

Flag : CTF{P1c4s0_woU1d_B3_pr0UD}