Messages

31

Mapping Tips/Guides / Day 2

« on: November 15, 2012, 06:09:45 pm »

Yesterday we got the map format decoded. While I can visualize more of the maps the same way, let's go ahead and try to figure out the tile format. Now, Xargon is a 256 colour VGA game, so we're going to need a colour palette. We also need to know what dimension of tile we are looking for. Both of these goals can be accompished by taking a simple screenshot (via Dosbox for me):



Looks like 16 x 16 tiles to me. Looking through the Xargon directory, GRAPHICS.XR1 is the most likely candidate for containing tile information. With the lack of a better idea, I'm going to just assume it contains a whole bunch of RAW 16 x 16 images in order. With a filesize of 697555, and each tile taking up 16*16 = 256 bytes, that is about 2724.8 tiles. Let's cook up a quick looping RAW image importer with PIL and see what we get:

Code: [Select]

import struct, sys, os
from PIL import Image

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargongraphics.py [Graphics File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            filesize = os.path.getsize(filename)
            graphicsfile = open(filename, 'rb')

            # Use the screenshot to grab the proper colour palette:
            palimage = Image.open('screeny.png')
            palette = palimage.getpalette()

            for tilenum in range(filesize/(16*16)):
                tile = Image.fromstring("P", (16, 16),
                    graphicsfile.read(16*16))
                tile.putpalette(palette)
                tile.save(os.path.join('output', '{:04}.png'.format(tilenum)) )

            # Since the file does not evenly align with the tiles we are
            # attempting to read, explicitly re-read the very end of the
            # file as a tile.

            graphicsfile.seek(filesize - (16*16) )
            tile = Image.fromstring("P", (16, 16),
                graphicsfile.read(16*16))
            tile.putpalette(palette)
            tile.save(os.path.join('output', 'last.png') )

And the results are:


Not a horrible assumption. There are some properly decoded (be it shifted) tiles among there. But there are also some images that look misaligned (i.e. not 16 x 16), and some random pixels (which imply headers). There's also a repeating pattern at the top of the file with random pixels that may be an overall file header (i.e. Images 0000 and 0002). Zooming the image shows that each record appears to be 4 bytes in length.

Scrolling through each of these 4-byte regions and looking at the decoding panel in my hex editor, each region appears to be a simple 32-bit integer, and each number is larger than the previous. (example below)



Let's decode those numbers. I locally commented out the image code and am excluding it from below for brevity. Counting the bytes in the Hex editor, it appears to end at offset 0xF8 for a total of 62 records. Also note that pdb.set_trace() in the below example invokes the python debugger where we can inspect data interactively. This also saves me the trouble writing a print statement for debug data.

Code: [Select]

import struct, sys, os, pdb
from PIL import Image

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargongraphics.py [Graphics File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            filesize = os.path.getsize(filename)
            graphicsfile = open(filename, 'rb')

            header = '<62L'

            headerdata = struct.unpack(header,
                graphicsfile.read(struct.calcsize(header)) )

            pdb.set_trace()

Running that gives us the the python debugger prompt:

Code: [Select]

zerker@Iota:/data/GameFiles/Dos/dosroot/Xargon$ python xargongraphics.py GRAPHICS.XR1
> /data/GameFiles/Dos/dosroot/Xargon/xargongraphics.py(28)<module>()
-> for filename in sys.argv[1:]:
(Pdb) headerdata

At which we can simply type the name of the variable we want to inspect to see its content:

Code: [Select]

(Pdb) headerdata
(0, 768, 9359, 14366, 15087, 21646, 22560, 54601, 61481, 68227, 74973, 87197, 94720, 98099, 101737, 124062, 131326, 140144, 148185, 155708, 160382, 163761, 171284, 178030, 197467, 204731, 214326, 235317, 253718, 262277, 266692, 296888, 305411, 309719, 321156, 325931, 343340, 353794, 363293, 374943, 395189, 407614, 0, 415826, 433431, 439659, 0, 451922, 473916, 0, 481227, 485490, 488542, 491213, 554159, 563460, 566188, 597635, 629082, 650040, 662219, 664428)

Keep in mind the file size of this graphics file is 697555. The last number is close, but does not exceed this number. This implies to me that these are all file offsets, probably delimiting the start of a region of data in the file. Since image 0002 also looks like header data, let's look at that too:



Well, it's smaller. Suspiciously looks like half as small, in fact. Let's grab that too and decode it as a series of 16 bit integers.

Code: [Select]

import struct, sys, os, pdb
from PIL import Image

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargongraphics.py [Graphics File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            filesize = os.path.getsize(filename)
            graphicsfile = open(filename, 'rb')

            header = '<62L'

            headerdata = struct.unpack(header,
                graphicsfile.read(struct.calcsize(header)) )

            graphicsfile.seek(0x200)

            header2 = '<62H'

            headerdata2 = struct.unpack(header2,
                graphicsfile.read(struct.calcsize(header2)) )

            pdb.set_trace()

And compare the two headers:

Code: [Select]

(Pdb) headerdata
(0, 768, 9359, 14366, 15087, 21646, 22560, 54601, 61481, 68227, 74973, 87197, 94720, 98099, 101737, 124062, 131326, 140144, 148185, 155708, 160382, 163761, 171284, 178030, 197467, 204731, 214326, 235317, 253718, 262277, 266692, 296888, 305411, 309719, 321156, 325931, 343340, 353794, 363293, 374943, 395189, 407614, 0, 415826, 433431, 439659, 0, 451922, 473916, 0, 481227, 485490, 488542, 491213, 554159, 563460, 566188, 597635, 629082, 650040, 662219, 664428)
(Pdb) headerdata2
(0, 8591, 5007, 721, 6559, 914, 32041, 6880, 6746, 6746, 12224, 7523, 3379, 3638, 22325, 7264, 8818, 8041, 7523, 4674, 3379, 7523, 6746, 19437, 7264, 9595, 20991, 18401, 8559, 4415, 30196, 8523, 4308, 11437, 4775, 17409, 10454, 9499, 11650, 20246, 12425, 8212, 0, 17605, 6228, 12263, 0, 21994, 7311, 0, 4263, 3052, 2671, 62946, 9301, 2728, 31447, 31447, 20958, 12179, 2209, 25423)

Interesting. Record sizes maybe? Let's compare differences between offsets for a few adjacent regions to the second set of data to test this theory:

Code: [Select]

(Pdb) headerdata[2]-headerdata[1]
8591
(Pdb) headerdata[3]-headerdata[2]
5007
(Pdb) headerdata[4]-headerdata[3]
721

Perfect match. I'm convinced. Now, going by the debug images I already generated, the first region appears to be a whole lot of blue and will be difficult to determine on its own. Let's start with the second region, at offset 9359 (0x248F hex). Since each picture is 256 bytes (0x100 hex), that puts towards the end of picture 0x24 (36 decimal). Hrm, still too much blue. Let's jump ahead until we get to the images. The next is 14366 -> image 56, which is still blue, but the following is 15087 -> 58 which starts to look like something. It also puts is one pixel before the last row of the image, which looks like another header. Since we're offset by 1, and the next image matches the last pixel in this image, I suspect the header is 16 bytes. I'll just copy the bytes from the hex editor:
24 01 00 D0 06 10 0D 90 19 08 04 00 08 10 00 10

Now we need to guess the bit size of each field in this header. Remember that we are little endian. The first four bytes are:
24 01 00 D0

If we think this is 32 bits, that's either -805306076 (signed) or 3489661220 (unsigned), or -8.590234E+09 (floating). None of those are particularly likely (and floats aren't very common for old DOS games in general).

16 bits yields 292, while 8 bits yield 36. Both are possible.

If we go with 36, 01 is unused, and is obviously just a simple 1. Let's convert the whole sequence to simple 8 bit decimal numbers and see if it makes sense:
36 1 0 208 6 16 13 144 25 8 4 0 8 16 0 16

And as 16 bits (unsigned):
292 53248 4102 36877 2073 4 4104 4096

I think it's likely we have either an 8 bit sequence or some sort of hybrid.

I'm scratching my head, so let's get another sample. The next few sample images look like noise, so they may be some other form of data, or just unrecognizably misaligned. Let's jump ahead to the really recognizable tiles around image 293. 293 is offset 75008, and from above, the closest record starts at 74973 (image 292, pixel 221). Looks like another 16 byte header to me. Hex values are:

2F 01 00 7C 0C 3C 18 BC 2F 08 04 00 10 10 00 0F
Decimal:
47 1 0 124 12 60 24 188 47 8 4 0 16 16 0 15

Gah, let's just grab them all and decode this way, then turn it into a CSV.

Code: [Select]

import struct, sys, os, pdb, csv
from PIL import Image

def imageheader(fileref, offset):
    fileref.seek(offset)
    headerstruct = '<16B'
    return struct.unpack(headerstruct,
        fileref.read(struct.calcsize(headerstruct)) )


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargongraphics.py [Graphics File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            filesize = os.path.getsize(filename)
            graphicsfile = open(filename, 'rb')

            header = '<62L'

            headerdata = struct.unpack(header,
                graphicsfile.read(struct.calcsize(header)) )

            graphicsfile.seek(0x200)

            header2 = '<62H'

            headerdata2 = struct.unpack(header2,
                graphicsfile.read(struct.calcsize(header2)) )

            # Create the header records using list comprehension
            imageheaders = [imageheader(graphicsfile, offset) for offset in headerdata if offset > 0]

            # Save as a CSV
            with open('debug.csv', 'wb') as csvfile:
                writer = csv.writer(csvfile)
                writer.writerows(imageheaders)

[code]
128 1   0   0   10  0   18  0   34  2   1   0   8   8   0   1
128 1   0   0   8   0   14  0   26  2   1   0   6   6   0   1
10  1   0   200 0   104 1   168 2   2   1   0   8   8   0   3
36  1   0   208 6   16  13  144 25  8   4   0   8   16  0   16
1   1   0   196 0   132 1   4   3   8   4   0   64  12  0   0
38  1   0   186 31  224 62  32  125 8   0   0   24  40  0   0
27  1   0   28  6   204 11  44  23  8   4   0   8   16  0   120
25  1   0   164 6   228 12  100 25  8   4   0   16  16  0   176
25  1   0   164 6   228 12  100 25  8   4   0   16  16  0   16
47  1   0   124 12  60  24  188 47  8   4   0   16  16  0   15
28  1   0   112 7   112 14  112 28  8   4   0   16  16  0   16
12  1   0   48  3   48  6   48  12  8   4   0   16  16  0   16
13  1   0   116 3   180 6   52  13  8   4   0   16  16  0   250
86  1   0   216 22  88  44  88  87  8   4   0   16  16  0   250
27  1   0   44  7   236 13  108 27  8   4   0   16  16  0   136
33  1   0   196 8   4   17  132 33  8   4   0   16  16  0   16
30  1   0   248 7   120 15  120 30  8   4   0   16  16  0   250
28  1   0   112 7   112 14  112 28  8   4   0   16  16  0   109
17  1   0   132 4   196 8   68  17  8   4   0   16  16  0   25
12  1   0   48  3   48  6   48  12  8   4   0   16  16  0   23
28  1   0   112 7   112 14  112 28  8   4   0   16  16  0   17
25  1   0   164 6   228 12  100 25  8   4   0   16  16  0   34
74  1   0   168 19  40  38  40  75  8   4   0   16  16  0   247
27  1   0   44  7   236 13  108 27  8   4   0   16  16  0   195
36  1   0   144 9   144 18  144 36  8   4   0   16  16  0   250
80  1   0   64  21  64  41  64  81  8   4   0   16  16  0   19
70  1   0   152 18  24  36  24  71  8   4   0   16  16  0   157
32  1   0   128 8   128 16  128 32  8   4   0   16  16  0   23
16  1   0   64  4   64  8   64  16  8   4   0   16  16  0   250
77  1   0   221 30  28  62  216 119 8   0   0   32  28  0   64
36  1   0   80  8   16  21  144 31  8   0   0   12  20  0   0
7   1   0   12  4   252 7   220 15  8   0   0   24  24  0   0
12  1   0   24  11  20  22  208 43  8   0   0   32  28  0   64
8   1   0   224 3   160 7   32  15  8   0   0   32  15  0   21
18  1   0   220 17  112 35  152 70  8   0   0   38  25  0   0
37  1   0   90  10  116 20  172 39  8   0   0   32  16  0   0
36  1   0   56  9   80  18  48  35  8   0   0   16  16  0   0
7   1   0   64  11  0   24  172 44  8   0   0   60  68  0   0
27  1   0   129 20  68  45  192 80  8   0   0   36  22  0   0
30  1   0   195 12  44  25  164 49  8   0   0   30  30  0   0
31  1   0   60  8   252 15  124 31  8   4   0   16  16  0   18
18  1   0   40  17  8   34  200 67  8   0   0   40  24  0   0
23  1   0   28  6   220 11  92  23  8   4   0   16  16  0   120
8   1   0   128 12  160 27  160 49  8   0   0   34  44  0   0
25  1   0   8   22  68  46  244 86  8   0&nbs

32

Mapping Tips/Guides / PC Game Hacking and Mapping Tutorial: Xargon

« on: November 14, 2012, 06:13:37 pm »

After finishing ROTT and the Shadow Caster maps, I decided to tackle something much simpler, but also to document the process as I go along. That game is Xargon, which can be obtained at Classic Dos Games. According to that site, Allen Pilgrim released the full version available as freeware with a very short license, so we will be able to map all three episodes. However, since I already have it on my computer, I will start with Episode 1. Note that the Source Code is released, so that is an option if I get stuck. However, to hopefuly make this tutorial more useful (and fun), I'm planning to do this as a black-box exercise. Obviously, if the game has a source code release, or is well documented, that can help reduce the guesswork.

As with Shadow Caster and ROTT, I will be using Python and the Python Imaging Library (PIL). I will also need a Hex editor to look through the game resources before digging in, so Bless is my editor of choice. Windows users will obviously need something else.

Generally speaking, creating a map via hacking is composed of three phases:
Phase 1: Figure out the map format
Phase 2: Figure out the image data format(s)
Phase 3: Put it all together and make the map (this the real meat of it)

Phases 1 and 2 can be done in either order, but I will start in the order above.

So, the map format. Xargon is split into a number of different files, and I'm going to take a wild guess that the BOARD_##.XR1 files are the maps. Let's look at one in a hex editor:



The first thing to notice is there's definately a repeating pattern. The second thing is that it looks like a 16 bit pattern. This can mean either 2 individual bytes per location, or a single 16 bit number per location. Either way, we know the size of a location, but not the dimensions of the map. We also know that there is no header, because the pattern starts immediately at the top of the file. Given the nature of the game, we will start with the assumption that the game map is simply a direct listing of tiles without any grouping, compression or other tricks. So, let's take the file size and figure out how many tiles we have total:

19255 bytes / 2 = 9627.5. This already tells us that we must have some sort of footer that isn't part of the map data. Scrolling down to the bottom of the file confirms this, as the map ends in the string "TOO LATE TO TURN BACK!". However, the footer is unlikely to be a huge portion of the file, so let's ignore it for now. Taking the square root of the file size gives us the dimensions if the game used square levels: 98.11. This gives us an order-of-magnitude to guess for. I know that the maps aren't square, but let's run with this and see where it gets us.

The next step is to visualize the map to see if we guessed correctly. 8-bit numbers are easiest, as we can go direct to grayscale. 16 bit numbers present us with two options: if we think the numbers present different information, we can try two parallel grayscale images. If we think it's just one 16 bit tile number, we should just pick two out of the three RGB channels and generate a colour image.  I'm going with the latter option initially. I'm also going to cheat a little bit and copy some boilerplate code from my other scripts as far as grabbing the input file and checking for # of input parameters. Here's the basic visualizer script:

Code: [Select]

import struct, sys
from PIL import Image

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargonmap.py [Map File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            mapfile = open(filename, 'rb')
            outname = filename + '.png'

            # Load the map data as a 98 x 98 array of 2-byte positions:
            # This will be switched to proper 16 bit numbers when we
            # actually want to start generating the tile map.
            # struct
            pattern = '<{}B'.format(98*98*2)

            mapdata = struct.unpack(pattern,
                mapfile.read(struct.calcsize(pattern)) )
            mapfile.close()

            # Turn the map data into a list of 3-byte tuples to visualize it.
            # Start by pre-creating an empty list of zeroes then copy it in
            visualdata = [None] * (98*98)
            for index in range(98*98):
                visualdata[index] = (mapdata[index * 2], mapdata[index * 2 + 1], 0)

            # Tell PIL to interpret the map data as a RAW image:
            mapimage = Image.new("RGB", (98, 98) )
            mapimage.putdata(visualdata)
            mapimage.save(outname)

'<{}B' means a little endian pattern of {} bytes, where the actual number is filled in by the format call. Refer to the Python Struct module documention for more information on these strings.

So we run it and get:



That's cute. Obviously wrong too, but there's a clear pattern shift to the image. We should be able to figure out exactly how far we are off on each row. Opening in GIMP and counting the pixel shift shows that we appear to repeat every 64 pixels. Taking our original dimensions, and using 64 as one dimension yields 9627.5/64 = 150.4 for the other. Some of that is going to be footer, but we should be able to clearly see the breakdown when we finish. Our adjusted script becomes:

Code: [Select]

import struct, sys
from PIL import Image

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargonmap.py [Map File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            mapfile = open(filename, 'rb')
            outname = filename + '.png'

            # Load the map data as a 98 x 98 array of 2-byte positions:
            # This will be switched to proper 16 bit numbers when we
            # actually want to start generating the tile map.
            # struct
            pattern = '<{}B'.format(64*150*2)

            mapdata = struct.unpack(pattern,
                mapfile.read(struct.calcsize(pattern)) )
            mapfile.close()

            # Turn the map data into a list of 3-byte tuples to visualize it.
            # Start by pre-creating an empty list of zeroes then copy it in
            visualdata = [None] * (64*150)
            for index in range(64*150):
                visualdata[index] = (mapdata[index * 2], mapdata[index * 2 + 1], 0)

            # Tell PIL to interpret the map data as a RAW image:
            mapimage = Image.new("RGB", (64, 150) )
            mapimage.putdata(visualdata)
            mapimage.save(outname)

Yeilding:



That looks WAY better. But it looks sideways. And we can clearly see the garbage at the bottom is the footer and is not tile data (we'll have to figure it out later; I suspect it is item/monster placement). Opening in GIMP again shows that the map is only 128 pixels tall, so let's fix that dimension. Let's also rotate it -90 degrees.

Code: [Select]

import struct, sys
from PIL import Image

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargonmap.py [Map File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            mapfile = open(filename, 'rb')
            outname = filename + '.png'

            # Load the map data as a 98 x 98 array of 2-byte positions:
            # This will be switched to proper 16 bit numbers when we
            # actually want to start generating the tile map.
            # struct
            pattern = '<{}B'.format(64*128*2)

            mapdata = struct.unpack(pattern,
                mapfile.read(struct.calcsize(pattern)) )
            mapfile.close()

            # Turn the map data into a list of 3-byte tuples to visualize it.
            # Start by pre-creating an empty list of zeroes then copy it in
            visualdata = [None] * (64*128)
            for index in range(64*128):
                visualdata[index] = (mapdata[index * 2], mapdata[index * 2 + 1], 0)

            # Tell PIL to interpret the map data as a RAW image:
            mapimage = Image.new("RGB", (64, 128) )
            mapimage.putdata(visualdata)
            mapimage.rotate(-90).save(outname)

And the final result (for today) is:

33

Mapping Tips/Guides / Shadow Caster Isometric Mapper and Resource Extractor release

« on: November 14, 2012, 05:06:30 pm »

Just wanted to let everyone know that I have released the scripts I wrote to generate my Shadow Caster maps. If anyone is interested in my process, you can download the scripts and have a look. If you own Shadow Caster, you can also re-generate the maps yourself, or use the scripts to extract most of the images from the game.

I will also be doing a thread on mapping Xargon shortly with a bit more details as I go.

34

Mapping Tips/Guides / Re: Emulator for PC games

« on: November 14, 2012, 04:10:47 am »

Agreed, decoding map files is not for everyone. I'm an Engineer, so tinkering with stuff is a fun exercise for me. It does require some programming knowledge, recognizing common patterns, and some guesswork. However, I find it a lot more fun than piecing together a basic screenshot map. Since I gave the example of Xargon above, and only the map screens and level 1 of Xargon have actually been submitted to the site, I think I'll tackle mapping the rest of the (shareware) game next. I will start a thread once I start on Xargon and document the process from beginning to end.

35

Mapping Tips/Guides / Re: Emulator for PC games

« on: November 10, 2012, 03:44:40 pm »

Some old PC games have very simple data formats and tile-based maps. Sometimes it can be more fun to decode the format and fill in the result. For example, Jill of the Jungle and Xargon (old Epic games) map formats are simply 128 x 64 2D arrays with each number determining the image tile to use.

36

Mapping Tips/Guides / Re: Recommended programs to assist your mapping

« on: November 10, 2012, 03:41:21 pm »

For the few maps I've done so far, I find Python and the struct module are invaluable for decoding old DOS formats, so long as you can find documentation for the format in question. Combine that with the Python Imaging Library (PIL), and you can load RAW images, create images programatically, then save the result.

For the less hacking inclined, ImageMagick is awesome for automatically stitching, cropping, converting, resizing a bunch of images (and more!).

37

Map Gab / Re: Rise of the Triad Maps?

« on: November 10, 2012, 06:27:38 am »

It seemed like a good project at the time. I'd be interested to know if PNGGauntlet made much of a difference.

FYI: I have since released the tools that I wrote to generate these maps. These tools will let you generate your own Isometric maps of 3rd party ROTT maps (assuming you own a copy of ROTT), or extract resources from the ROTT wad file.