Messages

And here's today's post. Time for the final level(s) and ending! Stage 10 isn't too complicated. There's just another illusionary wall, and the "To Reactor" Sign. There's also some sort of unidentified pickup hidden behind an illusionary wall with the ID of 42:0. Since the yellow gem was 42:3, I'm going to take the educated guess that this is the green gem. I'll comment it as "TBC" in case I actually see one in Episodes 2 or 3.

Stage 10 done:


Now, the Reactor stage. First, it appears to have the wrong palette, so I'll fix that immediately. Second, it has these infinite floating robot spawners. I'm going to create a "composite" sprite containing a couple robots clustered together and see how that looks.

Code Select Expand


# Special case for 73, type 0. Variant 4 appears to be the pickup item.
# Other variants (all rendered invisible) appear to be:
# 1 : Flaming Face Jet (Down)
# 2 : Flaming Lava Jet (Up)
# 3 : Robot Spawner
self.addsprite(73, 0, variablesprite({
    1 : graphics.records[30].images[19],
    2 : graphics.records[30].images[19],
    3 : graphics.compositeimage((32, 32), [(0, 0, 59, 1),
       (16, 0, 59, 4), (8, 12, 59, 1)]),
    4 : graphics.semitransparent(
        graphics.records[37].images[0], 128)},
    field='variant'))


Reactor Stage (32) Done:


Ending sequence time. It looks like the ending has a special font-mode which actually uses the two colours for the fonts. Since my setup doesn't let me do that, I'm going to work-around this by drawing the text twice, once for each colour, to get a somewhat similar effect.  Right now it shows up as dark gray, which is wrong.

Code Select Expand


def draw(self, mappicture, objrec, mapdata):
    pen = ImageDraw.Draw(mappicture)

    if objrec.appearance == 8:
        # Simulate multi-colour appearance by creating a fake shadow effect
        pen.text((objrec.x, objrec.y), mapdata.getstring(objrec.stringref),
                font=self.font, fill=self.graphics.getcolour(14))
        pen.text((objrec.x-1, objrec.y), mapdata.getstring(objrec.stringref),
                font=self.font, fill=self.graphics.getcolour(6))
    else:
        pen.text((objrec.x, objrec.y), mapdata.getstring(objrec.stringref),
                font=self.font, fill=self.graphics.getcolour(objrec.appearance))




Not perfect, but a reasonable facsimile. It will do. With that, the shareware game is fully mapped. It just needs a few tweaks before it's ready for the site.

Oh, but there are a few sprites in the "demo" stages that we haven't identified. And, unfortunately, we CAN'T identify without playing the original versions of those stages. Since the demo stages are essentially identical to their original versions, there's no real reason to submit these.

day17.zip is available.

Looks like I forgot to post what I wrote up yesterday. Let me get that out of the way:

Another day, more maps. Today I will be filling in the gaps for the next couple levels in the game. The further we get, the fewer sprites need identification. We'll see when we identify them all .

Another map, another palette. I'm going to rename the palette images to start at 0 and generalize this a bit:

Code Select Expand


self.palette = {}
for i in range(6):
    palimage = Image.open('palimage{}.png'.format(i) )
    self.palette[i] = palimage.getpalette()


Stage 8 is done:



And Stage 9:



Not a big day today. I'll map the final stage of Episode 1 tomorrow, then get to some of the other tasks I've been putting off before submitting them for the site. Episode 2 will follow that. day16.zip is available.

Hello again. Today will mostly just be me identifying some more sprites, but I thought I'd fix one general bug I noticed yesterday (and no, not the string order in the STORY stage). It's the problem where sprites are drawn over text, contrary to the in-game appearance.



So what I'm going to do is split the text sprites and non-text sprites into two separate lists, and always draw the text after everything else. The main change is in the map loading:

Code Select Expand


self.objs = [objrecord(struct.unpack(objrecstruct,
    mapfile.read(struct.calcsize(objrecstruct)) ) )
    for i in range(numobjs)]

# Create separate sprite and text lists:
self.text = [obj for obj in self.objs if obj.sprtype in [6, 7] ]
self.sprites = [obj for obj in self.objs if obj.sprtype not in [6, 7] ]


And use these two lists for the map:

Code Select Expand


for objrecord in mapdata.sprites:
    sprites.drawsprite(self.mappicture, objrecord, mapdata)
for objrecord in mapdata.text:
    sprites.drawsprite(self.mappicture, objrecord, mapdata)


Also, I'm going to make a really minor change and make the static flame sprites on the menus screens variable to take into account each version:

Code Select Expand


# Menu Flame Jets:
self.addsprite(47, 0, variablesprite({
    6 : graphics.records[48].images[3],
    8 : graphics.records[48].images[4]},
    field='info'))


And here's the update:



And on a total whim, I decided to check the palette of the title screen and see if that's the correct palette for image record 53. Turns out it is!



Sorry folks, that's about all the excitement today. Now I just get to play the game some more and identify more sprites. Well, that's not true. I found out that invisible platforms can come in multiple variants. I just added one to support stage 6:

Code Select Expand


# Variant of Compound and semi-transparent for hidden platform(s)
self.addsprite(11, 0, variablesprite({
    6: graphics.semitransparent(
       graphics.compositeimage((32, 16), [(0, 0, 25, 14),
       (16, 0, 25, 15)]), 128),
    7: graphics.semitransparent(
       graphics.compositeimage((32, 16), [(0, 0, 51, 10),
       (16, 0, 51, 11)]), 128) }
     ))


Stage 6 complete:


I did the stages out of order, so going back to stage 5 doesn't result in anything extraordinary. It has another palette, so I'll add that to the list. It also has switches and toggle-able walls. I will hide the wall sprites and draw the switches. I'll add the switches and walls to my "maybe label the link" list. A few more trap and item types round out the list.

Stage 5 complete:


Stage 7 complete too. Nothing major here. I'll pencil in the stalactites as another possible sprite to mark with trigger identifiers, so we can tell when they would drop (and what causes them).



day15.zip is available.

Good evening. The next couple days are likely going to be fairly boring. We have most of the functionality implemented, so I'm going to spend the time playing the game and identifying sprites. Yesterday we noticed a problem with the sprites in the Story map. I'll likely need to get some debug data output to look into that further, so I'm going to defer that to the weekend when I can look into it more.

The first thing I'm going to do is identify any sprite I can find in the menu maps and the demos. Following that, I'll continue where I left off in the game on stage 5.

The story scene sprites all appear continuous, so I can populate all of those with a simple loop:

Code Select Expand


# Story Scenes:
for subtype in range(24):
    self.addsprite(85, subtype, sprite(graphics.records[56].images[subtype]))
    self.addsprite(86, subtype, sprite(graphics.records[57].images[subtype]))




Mapping for a bit, I found out that the Episode 3 demo has a slightly different colour palette, so I'll incorporate that.

Code Select Expand


if self.name.upper() in ['BOARD_03', 'BOARD_05', 'BOARD_06',
        'BOARD_07', 'BOARD_09', 'BOARD_10', 'INTRO', 'DEMO1', 'DEMO2']:
    graphics.changepalette(1)
elif self.name.upper() == 'DEMO3':
    graphics.changepalette(2)
else:
    graphics.changepalette(0)


Code Select Expand


palimage = Image.open('palimage3.png')
self.palette[2] = palimage.getpalette()


For Demo 1, I added an entry for floor/ceiling spears. However, I need to expand the variable sprite to allow offsets, because the floor image spear appears inside the floor. I also need to make the offets a list to match with the lookup.

Code Select Expand


# Ceiling Spear
self.addsprite(43, 0, variablesprite({
    0 : graphics.records[36].images[9],
    1 : graphics.records[36].images[12]},
    offsets={0: (0, 0), 1:(0, -4) },
    field='direction'))


Code Select Expand


class variablesprite(sprite):
    def __init__(self, imagelookup, contents=None, field='apperance', offsets=None):
        # Create a lookup of possible boxes
        self.types = imagelookup
        self.xoffs = 0
        self.yoffs = 0
        self.contents = contents
        self.offsets = offsets
        self.field = field

    def draw(self, mappicture, objrec, mapdata):
        # Pick the correct image then use the parent routine to draw the box
        self.image = self.types[objrec.__dict__[self.field]]
        if self.offsets != None:
            (self.xoffs, self.yoffs) = self.offsets[objrec.__dict__[self.field]]
        super(variablesprite, self).draw(mappicture, objrec, mapdata)

        # Place contents immediately above the current sprite
        if self.contents != None:
            mappicture.paste(self.contents, (objrec.x +self.xoffs,
                objrec.y +self.yoffs - self.contents.size[1]), self.contents)


I also notice that it looks like the flaming jet face has a hidden ice cream cone instead of the sprite that would say "this is a flaming jet face".



Since the face is in the background, I just need to recognize this case and simply not draw anything. Looking at the Object file, it appears that the flame jet instance has a 1 in what I called the "Direction" field, while the other two have 4. I'm going to rename that field to variant, and make a special variablesprite case for sprite 73, type 0 in the spritedb:

Code Select Expand


# Special case for 73, type 0. Variant 4 appears to be the pickup item.
# Other variants (all rendered invisible) appear to be:
# 1 : Flaming Face Jet (Down)
# 2 : Flaming Lava Jet (Up)
# 3 : TBC
self.addsprite(73, 0, variablesprite({
    1 : graphics.records[30].images[19],
    2 : graphics.records[30].images[19],
    3 : graphics.debugimage(73, 'T3', 16, 16),
    4 : sprite(graphics.semitransparent(
        graphics.records[37].images[0], 128))},
    field='variant'))


That works for most stages, but it looks like I somehow broke something in DEMO3:

Code Select Expand


Generating Map 'DEMO3'
Traceback (most recent call last):
  File "xargonmapper.py", line 68, in <module>
    mapper = xargonmapper(xargonimages, tiledata, themap)
  File "xargonmapper.py", line 49, in __init__
    sprites.drawsprite(self.mappicture, objrecord, mapdata)
  File "/data/Projects/Xargon/spritedb.py", line 196, in drawsprite
    self.sprites[objrec.sprtype][objrec.subtype].draw(mappicture, objrec, mapdata)
  File "/data/Projects/Xargon/spritedb.py", line 255, in draw
    super(variablesprite, self).draw(mappicture, objrec, mapdata)
  File "/data/Projects/Xargon/spritedb.py", line 226, in draw
    objrec.y +self.yoffs), self.image)
  File "/usr/lib/python2.7/dist-packages/PIL/Image.py", line 1079, in paste
    "cannot determine region size; use 4-item box"
ValueError: cannot determine region size; use 4-item box


That's a bit cryptic, so I think it's time to make my own exception handler to get more info.

Code Select Expand


def drawsprite(self, mappicture, objrec, mapdata):
    try:
        if objrec.sprtype not in self.sprites or \
                objrec.subtype not in self.sprites[objrec.sprtype]:
            self.addsprite(objrec.sprtype, objrec.subtype, sprite(
                self.graphics.debugimage(objrec.sprtype, objrec.subtype,
                objrec.width, objrec.height)))

        self.sprites[objrec.sprtype][objrec.subtype].draw(mappicture, objrec, mapdata)

        #if objrec.info != 0:
        #    self.drawlabel(mappicture, (objrec.x -8, objrec.y -8), str(objrec.info))
    except:
        print "Problem with Sprite {}, Type {}, Appearance {}, Variant {} at ({}, {})".format(
            objrec.sprtype, objrec.subtype, objrec.apperance, objrec.variant,
            objrec.x, objrec.y)
        traceback.print_exc()


FYI: traceback.print_exc() will print the standard stack trace you get without any exception handler.
When I run it again, I get:

Code Select Expand


Problem with Sprite 73, Type 0, Appearance 0, Variant 4 at (848, 640)
Traceback (most recent call last):
< Snip >

Problem with Sprite 73, Type 0, Appearance 0, Variant 4 at (240, 592)
Traceback (most recent call last):
< Snip >


Okay, so it's something about the NORMAL semi-transparent ice cream cone. Let's look at that line.

Code Select Expand


4 : sprite(graphics.semitransparent(
                graphics.records[37].images[0], 128))},


Gah, I'm declaring a sprite inside a variable sprite declaration! I should just be passing the semi-transparent image in directly. Fixing now.

And that's everything I can access from the main menu. I think I'll stop for today. day14.zip is available.

Now for something different. Today is font day!

For ROTT, I made my own font class, which contained an array of images that got pasted into the map. It worked, but this time I'd like to leverage the PIL ImageFont capabilities (especially for the re-colouring). Problem is, I need to get the font data into a format that PIL can use. I see two options here:
1) Look up the format for BDF or PCF font files, and create one
2) Figure out the underlying PIL font code and create a compatible implementation based on the in-game font.

Looking at the definition of BDF, that format doesn't look too bad, so I think that is the most likely avenue here. A side-effect of this is that we can actually use the Xargon font in BDF-supporting applications.

To get the font ready, I'm going to start a new file called xargonfontgen.py. This will create a BDF from both main fonts in the Xargon GRAPHICS file. To do this, we're going to need to convert the font images into 1-bit images. That said, we should get 256-colour exports of the font images to decide which "colours" are used. Let's update the xargongraphics save routines and main function:

Code Select Expand


xargonimages.save('Images')
xargonimages.save('OriginalImages', masked=False)


Code Select Expand


def save(self, outpath, masked=True):
    createpath(outpath)
    for recnum, record in enumerate(self.records):
        record.save(outpath, recnum, masked)


Code Select Expand


def save(self, outpath, recnum, masked=True):
    if self.numimages > 0:
        createpath(outpath)
        if (masked):
            for tilenum, tile in enumerate(self.images):
                tile.save(os.path.join(outpath, '{:02}-{:04}.png'.format(recnum, tilenum)) )
        else:
            for tilenum, tile in enumerate(self.origimages):
                tile.save(os.path.join(outpath, '{:02}-{:04}.png'.format(recnum, tilenum)) )


I actually worked a bit ahead on this yesterday, but knew I wasn't going to finish in one day. Astute followers may have noticed the above updates already in yesterday's zip file.

Looking at one of the font images, they appear to be effectively 3-bit images, using values of 1, 2 and 3 (not 0). Value 3 is transparent. From my observations, the menu text appears to use the two different colours, but all the in-map text (i.e. what we care about) does not.

Next, let's compare the images to an ASCII table to make sure they're in the same order... and they are. Same indices too, although the non-printable characters appear to be re-used. So let's start with a basic class that can convert to 1-bit images, but instead of writing the font at first, let's write some debug images to make sure we're categorizing things correctly.

Code Select Expand


from xargongraphics import createpath, imagefile
import sys, os
from PIL import Image

class xargonfont(object):
    @staticmethod
    def conv1bit(inimage):
        outimage = Image.new('1', inimage.size )
        imgdata = list(inimage.getdata())
        for index, pixel in enumerate(imgdata):
            imgdata[index] = 0 if pixel==3 else 1
        outimage.putdata(imgdata)
        return outimage

    def __init__(self, graphics, recnum):
        self.dimensions = graphics.records[recnum].images[0].size
        self.characters = [self.conv1bit(image) for image in graphics.records[recnum].origimages]

    def debugimages(self, outfolder):
        createpath(outfolder)
        for glyphnum, glyph in enumerate(self.characters):
            if glyphnum >= 32:
                glyph.save(os.path.join(outfolder, '{:02}.png'.format(glyphnum)) )


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargongraphics.py [Graphics File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            graphics = imagefile(filename)
            font1 = xargonfont(graphics, 1)
            font2 = xargonfont(graphics, 2)
            font1.debugimages('font1')
            font2.debugimages('font2')




Looks about right. Note that the PIL 1-bit image has 1 = white, 0 = black. We can ignore this. The BDF format has the reverse definition, and that's what we really want to match.

So here's what I came up with to write out the BDF file:

Code Select Expand


def createbdf(self, outname):
    with open(outname, 'w') as outfile:
        outfile.write("""STARTFONT 2.1
FONT {0}
SIZE {1[0]} 75 75
FONTBOUNDINGBOX {1[0]} {1[1]} 0 0
CHARS {2}\n""".format(self.name, self.size, 95))
        for charnum, char in enumerate(self.characters):
            if charnum >= 32 and charnum < 127:
                outfile.write("""STARTCHAR U+{0:04X}
ENCODING {0}
SWIDTH 0 0
DWIDTH {1[0]} 0
BBX {1[0]} {1[1]} 0 0
BITMAP\n""".format(charnum, self.size))
                for y in range(self.size[1]):
                    value = 0
                    for x in range(self.size[0]):
                        value = value + (char.getpixel((x,y)) << self.size[0]-x-1)
                    # Note: this will break if font width is > 8 bits!
                    outfile.write("{:02X}\n".format(value))
                outfile.write("ENDCHAR\n")
        outfile.write("ENDFONT\n")

Note that the alignment is a little bit funny because I'm using multi-line strings. Python's string format method really helps for filling in the data in the appropriate fields. For the meat of the function, I'm just combining each row of the image into a number by using add and bit shift. I'm sure there's a more efficient/pythonic way of doing it, but I only need this to work ONCE to get me a valid BDF file, then I can just keep using the output.

And here's what I get for the first two characters:

Code Select Expand


STARTFONT 2.1
FONT xargon-font1
SIZE 8 75 75
FONTBOUNDINGBOX 8 8 0 0
CHARS 95
STARTCHAR U+0020
ENCODING 32
SWIDTH 0 0
DWIDTH 8 0
BBX 8 8 0 0
BITMAP
00
00
00
00
00
00
00
00
ENDCHAR
STARTCHAR U+0021
ENCODING 33
SWIDTH 0 0
DWIDTH 8 0
BBX 8 8 0 0
BITMAP
18
3C
3C
18
18
00
18
00
ENDCHAR


Seems promising. Let's try this font out.

Code Select Expand


$ pilfont.py *.bdf
xargonfont1.bdf...
Traceback (most recent call last):
  File "/usr/bin/pilfont.py", line 48, in <module>
    p = BdfFontFile.BdfFontFile(fp)
  File "/usr/lib/python2.7/dist-packages/PIL/BdfFontFile.py", line 114, in __init__
    font[4] = bdf_slant[string.upper(font[4])]
IndexError: list index out of range

Uh oh. Now I need to debug someone ELSE's code. Well, more accurately, why my output is wrong and causing the code to crash. Anyhow, let's go open this up. The which command tells me pilfont.py is in /usr/bin/pilfont.py, so I'm going to open that up and find the line in question.

And that just points to the BdfFontFile class inside of PIL, so let's find that. Ah, it's processing the FONT line. I didn't think it would actually need any useful information from that. Let me try and update that to be more representative. It seems to me that it's just performing some adjustments to the name, so I should just need to make sure I have the right number of fields. I'll just grab the wikipedia example and code to match.

Code Select Expand


outfile.write("""STARTFONT 2.1
FONT -{0}-medium-r-normal--{1[0]}-160-75-75-c-80-us-ascii
SIZE {1[0]} 75 75
FONTBOUNDINGBOX {1[0]} {1[1]} 0 0
CHARS {2}\n""".format(self.name, self.size, 95))


That gets us past that block, but now we get: KeyError: 'FONT_ASCENT'. Well, we omitted FONT_ASCENT because the BDF spec says it's optional, and it doesn't really apply to us. Guess pilfont requires it. Let's just put it in and set it to the height with the DESCENT set to 0.

Code Select Expand


outfile.write("""STARTFONT 2.1
FONT -{0}-medium-r-normal--{1[0]}-160-75-75-c-80-us-ascii
SIZE {1[0]} 75 75
FONTBOUNDINGBOX {1[0]} {1[1]} 0 0
STARTPROPERTIES 2
FONT_ASCENT {1[1]}
FONT_DESCENT 0
ENDPROPERTIES
CHARS {2}\n""".format(self.name, self.size, 95))


Now, pilfont runs with no errors. Let's try out the new .pil files.

Code Select Expand


# Text sprites:
self.addsprite(6, 0, textsprite(ImageFont.load("font2.pil"), graphics))
self.addsprite(7, 0, textsprite(ImageFont.load("font1.pil"), graphics))


We can also take out the alignment corrections we added for the true-type font we picked inside the textsprite class.



Well, one font looks good, but the other is cut off. You know, I bet the BDF spec has left-aligned pixels instead of right-aligned pixels. This means I need to shift my 6-pixel font 2 images over 2 more pixels. For simplicity, I will simply update to align to an 8 bit number in my BDF export:

Code Select Expand


for x in range(self.size[0]):
    value = value + (char.getpixel((x,y)) << 7-x)




There we go. Font victory!

Now, just for fun, I'm going to use the built-in Xargon font for all my debug images and stop including DroidSans.

Oh, BTW, I just noticed additional files that actually contain map data (DEMO*.XR1, INTRO.XR1, STORY.XR1), so I'll be sure to include those too. Unfortunately, my clever hackery for string processing has malfunctioned on the STORY scene:



I'll have to keep that in mind, but I probably won't be changing anything there for a while. Also, the map ID isn't a good way to recognize alternate palettes, as some of the auxiliary files have different palettes but the same map id of 0. I'm going to switch over to filenames.

Code Select Expand


if self.name.upper() in ['BOARD_03', 'BOARD_05', 'BOARD_06',
        'BOARD_07', 'BOARD_09', 'BOARD_10', 'INTRO', 'DEMO1',
        'DEMO2', 'DEMO3']:
    graphics.changepalette(1)
else:
    graphics.changepalette(0)



FYI: day13.zip is available.

Good morning folks. When doing the switches and treasure boxes yesterday, I came to a realization. The column that I first attempted as the treasure box appearance actually seems to be the link between a switch and what it affects. Why don't we try drawing this identifier into the world to clearly indicate the effects of switches? If we don't like how it turns out, we can always disable it again.

To do this, I'm simply going to create a new method in the spritedb file to draw this onto the map. Since this is NOT debug text, we're going to want to draw this in a way that is more visible. I'm going to re-use the code from my Shadow Caster map to draw it first offset a couple times in black, then in white to get a bordered effect.

Code Select Expand


def drawsprite(self, mappicture, objrec, mapdata):
    if objrec.sprtype not in self.sprites or \
            objrec.subtype not in self.sprites[objrec.sprtype]:
        self.addsprite(objrec.sprtype, objrec.subtype, sprite(
            self.graphics.debugimage(objrec.sprtype, objrec.subtype,
            objrec.width, objrec.height)))

    self.sprites[objrec.sprtype][objrec.subtype].draw(mappicture, objrec, mapdata)

    if objrec.info != 0:
        self.drawlabel(mappicture, (objrec.x -8, objrec.y -8), str(objrec.info))


def drawlabel(self, mappicture, coords, text):
    # Draw the text 5 times to create an outline
    # (4 x black then 1 x white)
    pen = ImageDraw.Draw(mappicture)
    for offset, colour in [( (-1,-1), (0,0,0) ),
            ( (-1,1), (0,0,0) ),
            ( (1,-1), (0,0,0) ),
            ( (1,1), (0,0,0) ),
            ( (0,0), (255,255,255) )]:
        pen.text((coords[0] + offset[0], coords[1] + offset[1]),
            text, font=self.markupfont, fill=colour)




Well, it seems to show up in too many places. I'm going to comment it out for now, but let's keep it in the back of our minds. We may still want something similar, but only for specific scenarios. This also appears to provide links for doorways, which is always handy.

In any case, I'm just going to go ahead and identify items in stage 4 now. Though I'm getting really tired of going in and out of subfolders to try to find sprites. I'm going to flatten my export structure from the graphics file, and remove the record offset.

Code Select Expand


    def save(self, outpath):
        createpath(outpath)
        for recnum, record in enumerate(self.records):
            record.save(outpath, recnum)


Code Select Expand


def save(self, outpath, recnum):
    if self.numimages > 0:
        createpath(outpath)
        for tilenum, tile in enumerate(self.images):
            tile.save(os.path.join(outpath, '{:02}-{:04}.png'.format(recnum, tilenum)) )


The first few sprites weren't anything special, although I found the sprite number of an illusionary wall, which I made semi-transparent. However, I ran into the following location on the map:



Which doesn't contain any unknown sprites in my current map. This must be another object I'm not drawing. Let me comment out my hidden objects and find which one it is. But even turning off every identified sprite STILL doesn't draw anything in this location. It doesn't make any sense. I will finish mapping this stage, then double-check my map decoding to make sure I'm not missing any sprites somehow.

I also ran into ceiling sprites, which are apparently the same ID as floor spikes. Probably differentiated by the "colour" field. I'm going to go ahead and rename that field to appearance. Though looking at the objs file, it appears to be the next field one over. I'll tentatively call this one "direction" and update the variablesprite class to specify which field to look up. I'll have to use access the raw underlying python field dictionary to make it work, though.

Code Select Expand


class variablesprite(sprite):
    def __init__(self, imagelookup, contents=None, field='apperance'):
        # Create a lookup of possible boxes
        self.types = imagelookup
        self.xoffs = 0
        self.yoffs = 0
        self.contents = contents
        self.field = field

    def draw(self, mappicture, objrec, mapdata):
        # Pick the correct image then use the parent routine to draw the box
        self.image = self.types[objrec.__dict__[self.field]]
        super(variablesprite, self).draw(mappicture, objrec, mapdata)

        # Place contents immediately above the current sprite
        if self.contents != None:
            mappicture.paste(self.contents, (objrec.x +self.xoffs,
                objrec.y +self.yoffs - self.contents.size[1]), self.contents)


Code Select Expand


# Spikes:
self.addsprite(59, 0, variablesprite({
    0 : graphics.records[36].images[28],
    1 : graphics.records[36].images[32]},
    field='direction'))


Looking over the objects file, I figured out what was going on. Those bouncing ball traps reported themselves as having no size in the objects file! This malfunctioned by creating an invisible debug image. I'm going to go ahead and switch to always generate a debug image. For any sprites I did not want to draw, I'm going to redirect to record 36, image 28 instead (i.e. an empty sprite).

With that, stage 4 is complete.



day12.zip is available. I've moved the CSVs and flat maps into a sub-folder in order to better organize things.

Today I'm going to handle the slightly more complicated sprites we want to handle. First, here's a screenshot of the centipede monster we want to re-create from the individual segment sprites:



I count a head, six segments, then a tail. The map data tells me that the bounding box is 76 x 22, although the actual sprite dimensions vary from 16x20 (head) to 8x17 (segment) to 12x8 (tail). However, each segment appears to connect directly to the next with no padding on the sprite image itself, so that will make things easy. Adding up the required widths adds up to exactly 76, so there's some additional confirmation. Let's make up a method for making a composite sprite. I'm adding this to the graphics file, since it owns the original images. It could have also been added to the spritedb file, but it would need a copy of the graphics file, so that seems somewhat silly.

Code Select Expand


def compositeimage(self, dimensions, imgrequests):
    tempimage = Image.new("RGBA", dimensions)
    for (x, y, recnum, imgnum) in imgrequests:
        pasteimage = self.records[recnum].images[imgnum]
        tempimage.paste(pasteimage, (x, y), pasteimage)
    return tempimage


Code Select Expand


self.addsprite(52, 7, sprite(graphics.compositeimage((76, 22), [(0, 0, 52, 0),
    (16, 5, 52, 1), (24, 5, 52, 2), (32, 5, 52, 3), (40, 5, 52, 4),
    (48, 5, 52, 5), (56, 5, 52, 6), (64, 7, 52, 7)] )))


And that worked out well enough:


On to transparency. Again, I'll add this to the graphics file, although it does not really need to go there. It's going to be a static method that simply takes the image to make transparent and the desired alpha value. We're simply going to use the multiply channel operation to perform this:

Code Select Expand


@staticmethod
def semitransparent(inimage, alpha):
    alphaimage = Image.new("RGBA", inimage.size, (255, 255, 255, alpha))
    return ImageChops.multiply(inimage, alphaimage)


And expanding our pickup population loop:

Code Select Expand


# Pickups appear to be in the same order as their corresponding record.
# There are two types of pickups: normal and hidden.
for subtype in range(24):
    self.addsprite(33, subtype, sprite(graphics.records[37].images[subtype]))
    self.addsprite(73, subtype, sprite(graphics.semitransparent(
        graphics.records[37].images[subtype], 128) ))


Looking good:


Stage 2 is done now. On to stage 3.

Well now. Stage 3 had a black background (that sometimes flashes with lightning). That is a far cry from the blue the map generates for it. Taking a look at the colour palette of the screenshot, and it is indeed different. Board 6 and 7 also look like they should have black backgrounds. However, looking at the map data doesn't seem to imply anything different between those maps and the maps with blue backgrounds. I think I'm just going to have to by the map number for this.

Oh yeah, and re-architect my graphics and sprite routines to allow switching palettes. Piece of cake, right?

Also, we have another problem in stage 3:


Alternate present boxes! Back to the object csv:

Code Select Expand


26  864     800 0   0   16  16  0   0   0   2   0   0   0   0   0
26  560     784 1   0   16  16  2   0   0   3   0   0   0   0   0
26  608     768 1   0   16  16  1   0   0   3   0   0   0   0   0
26  496     720 1   0   16  16  2   0   0   3   0   0   0   0   0
26  528     736 1   0   16  16  2   0   0   3   0   0   0   0   0
26  128     112 0   0   16  16  13  0   0   4   0   0   0   0   0
26  112     128 0   0   16  16  11  0   0   4   0   0   0   0   0
26  144     112 0   0   16  16  11  0   0   4   0   0   0   0   0
26  160     96  0   0   16  16  11  0   0   4   0   0   0   0   0
26  176     128 0   0   16  16  11  0   0   4   0   0   0   0   0
26  192     112 0   0   16  16  13  0   0   4   0   0   0   0   0
26  864     272 0   -1  16  16  0   0   0   0   0   0   0   0   0
26  1936    144 0   -1  16  16  0   0   0   0   0   0   0   0   0
26  1824    896 0   -1  16  16  11  0   0   0   0   0   0   0   0
26  1472    896 1   -1  16  16  2   0   0   0   0   0   0   0   0
26  1856    528 0   -1  16  16  11  0   0   0   0   0   0   0   0
26  1376    608 1   0   16  16  2   0   0   5   0   0   0   0   0
26  1328    624 1   0   16  16  13  0   0   5   0   0   0   0   0
26  1280    608 1   0   16  16  13  0   0   5   0   0   0   0   0
26  1264    624 1   0   16  16  2   0   0   5   0   0   0   0   0
26  1168    624 1   0   16  16  2   0   0   5   0   0   0   0   0
26  1040    736 1   0   16  16  2   0   0   5   0   0   0   0   0
26  1936    992 0   -1  16  16  0   0   0   0   0   0   0   0   0
26  272     944 0   -1  16  16  0   0   0   0   0   0   0   0   0
26  1312    992 1   -1  16  16  4   0   0   0   0   0   0   0   0
26  704     816 1   0   16  16  12  0   0   3   0   0   0   0   0


I'm going to guess it's that column that goes 2, 3, 4, 0, 5, 3. To avoid totally re-architecting my sprite algorithm for ONE special case, I'm going to create a new type of sprite for treasure boxes. And yank the contents handling from the normal sprite class, since it doesn't get used in any other context.

Code Select Expand


class treasuresprite(sprite):
    def __init__(self, graphics, contents):
        # Create a lookup of possible boxes
        self.types = {0 : graphics.records[37].images[25],
            1 : graphics.debugimage('T', 1, 16, 16),
            2 : graphics.debugimage('T', 2, 16, 16),
            3 : graphics.records[37].images[27],
            4 : graphics.debugimage('T', 4, 16, 16),
            5 : graphics.debugimage('T', 5, 16, 16)
            }
        self.xoffs = 0
        self.yoffs = 0
        self.contents = contents

    def draw(self, mappicture, objrec, mapdata):
        # Pick the correct image then use the parent routine to draw the box
        self.image = self.types[objrec.info]
        super(treasuresprite, self).draw(mappicture, objrec, mapdata)

        # Place contents immediately above the current sprite
        mappicture.paste(self.contents, (objrec.x +self.xoffs,
            objrec.y +self.yoffs - self.contents.size[1]), self.contents)


Well, that works and all, but it misidentifies several treasure boxes. Hrm, comparing map 1 to this map, it appears that map 1 always uses '3' in the 'colour' field, while this map uses 0 and 1. Let's assume it's the colour field instead and re-do this.



Fixed that. Next is the palette thing. To do this, we need to defer the masking operation in the graphics file. We should store the images in index format initially, then create the masked version on-demand. Since we already have other functions directly referencing graphics.records[n].images[m], we should store the raw data in a different location:

Code Select Expand


tile = Image.fromstring("P", (width, height),
    self.filedata.read(width*height))
tile.putpalette(palette)
self.origimages.append(tile)
self.images.append(self.maskimage(tile))


Then we need a method of switching palettes:

Code Select Expand


def changepalette(self, palette):
    for imagepos, image in enumerate(self.origimages):
        image.putpalette(palette)
        self.images[imagepos] = self.maskimage(image)


Finally, some control at the graphics file level to switch between a choice of palettes. I will also add a check to save time by not switching to the same palette if it is already loaded:

Code Select Expand


def changepalette(self, palnum):
    if self.activepal != palnum:
        self.activepal = palnum
        for record in self.records:
            record.changepalette(self.palette[self.activepal])


And don't forget the getcolour method:

Code Select Expand


def getcolour(self, index):
    return tuple(self.palette[self.activepal][index*3:index*3+3])


Then we just need to change palettes for each map. We could go by the map name, but I noticed the first byte in the "unknown" region of the map data appears to be the map number. I'm going to go by that.

Code Select Expand


if mapdata.mapnum in [3, 6, 7, 10]:
    graphics.changepalette(1)
else:
    graphics.changepalette(0)
sprites = spritedb(graphics)


And it works as expected. Now we don't have the garish blue background in the dark levels. Time to continue identifying sprites in level 3.

Well, I ran into another interesting thing. Sprite 12 can sometimes be visible! We need to determine what decides this and adjust accordingly. Checking the object CSV, it looks like it has a 1 in what I called the "colour" column. Looks like I should expand my treasure box sprite into something that can also handle other similar variable sprites. I will then have to make the contents optional again. Let's see how that turns out:

Code Select Expand


class variablesprite(sprite):
    def __init__(self, imagelookup, contents=None):
        # Create a lookup of possible boxes
        self.types = imagelookup
        self.xoffs = 0
        self.yoffs = 0
        self.contents = contents

    def draw(self, mappicture, objrec, mapdata):
        # Pick the correct image then use the parent routine to draw the box
        self.image = self.types[objrec.colour]
        super(variablesprite, self).draw(mappicture, objrec, mapdata)

        # Place contents immediately above the current sprite
        if self.contents != None:
            mappicture.paste(self.contents, (objrec.x +self.xoffs,
                objrec.y +self.yoffs - self.contents.size[1]), self.contents)


The actual lookup gets pulled out into the sprite db prior to creating the treasure boxes:

Code Select Expand


# Treasures (+ contents)
treasurelookup = {0 : graphics.records[37].images[24],
    1 : graphics.records[37].images[25],
    2 : graphics.debugimage('T', 2, 16, 16),
    3 : graphics.records[37].images[27] }

for (sprtype, subtype, crecnum, cimagenum) in [
        (26, 0, 37, 33), # Health
        (26, 1, 37, 2), # Grapes
        (26, 2, 37, 6), # Cherry
        (26, 4, 37, 14), # Orange
        (26, 11, 30, 28), # Emerald
        (26, 12, 48, 2), # Nitro!
        (26, 13, 36, 29) # Empty
        ]:
    self.addsprite(sprtype, subtype, variablesprite(treasurelookup,
        contents=graphics.records[crecnum].images[cimagenum]))


And the switches. FYI: 30, 19 is an empty sprite

Code Select Expand


# Switches:
self.addsprite(12, 0, variablesprite({
    0 : graphics.records[30].images[19],
    1 : graphics.records[51].images[0]}))


Oh yeah, and there's a series of hidden platforms that are exposed by this switch. Let's create them as a composite AND semi-transparent sprite:

Code Select Expand


self.addsprite(11, 0, sprite(graphics.semitransparent(
    graphics.compositeimage((32, 16), [(0, 0, 25, 14),
    (16, 0, 25, 15)]), 128) ))


And here's how the switch and hidden platforms look:


With that, and a few more sprites identified, stage 3 is done. day11.zip is available.

Good evening. Time to tackle the two enhancements we wanted yesterday. Enhancement #1 is for present contents. For these I'm just going to expand the capabilities of the basic sprite class and not even make a separate class. I will simply add an optional parameter for the content sprite. When drawing the sprite, if contents are assigned, they will be drawn immediately above the sprite itself.

Code Select Expand


class sprite(object):
    def __init__(self, image, xoffs=0, yoffs=0, contents=None):
        self.image = image
        self.xoffs = xoffs
        self.yoffs = yoffs
        self.contents = contents

    def draw(self, mappicture, objrec):
        # When pasting masked images, need to specify the mask for the paste.
        # RGBA images can be used as their own masks.
        mappicture.paste(self.image, (objrec.x +self.xoffs,
            objrec.y +self.yoffs), self.image)

        if contents != None:
            # Place contents immediately above the current sprite
            mappicture.paste(self.contents, (objrec.x +self.xoffs,
                objrec.y +self.yoffs - self.contents.size[1]), self.contents)



And minor changes to the database:

Code Select Expand


# Presents have contents:
for (sprtype, subtype, recnum, imagenum, crecnum, cimagenum) in [
        (26, 2, 37, 27, 37, 6), # Treasure (Cherry)
        (26, 0, 37, 27, 37, 33) # Treasure (Health)
        ]:
    self.addsprite(sprtype, subtype, sprite(graphics.records[recnum].images[imagenum],
        contents=graphics.records[crecnum].images[cimagenum]))




That was easy. Now for text. There are two approaches we can use for text:
1) Use the raw text images we have to reproduce the in-game text
2) Find a close match and just go with it

That said, what we got directly out of the reference file appears a bit strange and would need some post-processing to get it to appear correct. The second option is certainly simpler, and I will go with that for now. As long as I can find a fixed-width font with reasonably similar weight and size, it should be pretty equivalent. For reference, here is the direct font data from the GRAPHICS file:



Stage 1 is easy, but we need to figure out the general solution for picking the correct text string. For that, we will refer to stages 3 and 33 (aka the ending).

Let me include the records for object 7 from each stage together below (stage # is the first column). I will also number the columns from the original file as a zero-based index along top.

Code Select Expand


    0   1       2   3   4   5   6   7   8   9   10  11  12  13  14  15  16
1   7   1584    788 9   -1  184 8   0   0   0   0   0   4   55  68  0   0
3   7   1824    560 6   -1  112 8   0   0   0   0   0   4   95  72  0   0
3   7   1856    576 6   -1  32  8   0   0   0   0   0   4   94  72  0   0
33  7   1104    36  7   -1  160 8   0   0   0   0   0   4   71  72  0   0
33  7   1440    36  7   -1  160 8   0   0   0   0   0   4   69  72  0   0
33  7   1120    130 7   -1  120 8   0   0   0   0   0   4   55  72  0   0
33  7   1456    130 7   -1  120 8   0   0   0   0   0   4   44  72  0   0
33  7   112     484 7   -1  136 8   0   0   0   0   0   4   31  72  0   0
33  7   112     578 7   -1  120 8   0   0   0   0   0   4   25  72  0   0
33  7   144     708 7   -1  56  8   0   0   0   0   0   4   24  72  0   0
33  7   80      48  6   -1  176 8   0   0   0   0   0   4   5   72  0   0
33  7   112     64  6   -1  112 8   0   0   0   0   0   4   3   72  0   0
33  7   112     96  7   -1  112 8   0   0   0   0   0   4   1   72  0   0
33  7   512     820 8   -1  120 8   0   0   0   0   0   4   250 71  0   0


We've already identified indices 0 to 2 and 5 - 7, although the obvious choice of the subtype does not appear to be used to select the specific string to be displayed. The only one that appears different for each string is column 13, but that doesn't make much sense on its own. That said, I think column 3 might be the colour index, of the first 16 colours in the palette. 9 is blue, 7 is white, 6 is yellow, 8 is dark gray.

Hrm, giving it a bit more of a look, it's possible that it might be part of a 16-bit number. The wrap around from 1 72 to 250 71 seems to imply that. Let me fix my decoding and try again. Here are just the numbers, (+ some in hex)

Code Select Expand


1   17463   0x4437
3   18527   0x485F
3   18526   0x485E
33  18503
33  18501
33  18487
33  18476
33  18463
33  18457
33  18456
33  18437
33  18435
33  18433
33  18426


I don't get it. I'm just going to have to hack something together. The ONLY pattern I think I can figure out is that these numbers are decreasing compared to the order of the strings. I also noticed that apparently object #17 also uses strings, as does object 6. Object 17 appears to be for internal use, so I won't actually draw those, but object 6 appears to be another string region (different font maybe?). Either way, here's what my hacked up approach will be:
1) Look for all numbers in this column, and collect them in a list in descending order.
2) When I have a string to grab, I will pull the string that corresponds to the order within the strings array corresponding to this number.

That should work for maps 1 and 3, but I have no idea how well it will sort itself out for map 33. At least it will get us un-stuck for now. First, let's add the new fields to the record:

Code Select Expand


class objrecord(object):
    def __init__(self, record):
        self.rawdata = record
        (self.sprtype, self.x, self.y, self.colour) = record[0:4]
        (self.width, self.height, self.subtype) = record[5:8]
        self.stringref = record[13]


Now, let's get that lookup table implemented:

Code Select Expand


# String reference lookup table. This is a bit of a hack for now.
# Sort all known string references in reverse order:
self.stringlookup = [record.stringref for record in self.objs if record.stringref > 0]
self.stringlookup.sort(reverse=True)


And a lookup method to get the correct string:

Code Select Expand


def getstring(self, stringref):
    strindex = self.stringlookup.index(stringref)
    return self.strings[strindex]


To get the right text colour, I will need a palette lookup method for the graphics object:

Code Select Expand


def getcolour(self, index):
    return tuple(self.palette[index*3:index*3+3])


To be able to do the text lookup, I need to pass a reference to the map into my sprite draw routine. Then I need to actually write the text sprite class to take advantage of all this new infrastructure. For the font, I'll just go with DroidSansMono for now. I'll pick 10 point font for object 7 and 8 point font for object 6. Here's how the text sprite class ended up:

Code Select Expand


class textsprite(sprite):
    def __init__(self, font, graphics):
        self.font = font
        self.graphics = graphics

    def draw(self, mappicture, objrec, mapdata):
        pen = ImageDraw.Draw(mappicture)
        pen.text((objrec.x, objrec.y), mapdata.getstring(objrec.stringref),
                font=self.font, fill=self.graphics.getcolour(objrec.colour))


More importantly, it works!



And even works for Board 33!



But the font is a bit too thin. Let me see if I can find a monospaced font that has a dedicated bold version. I'll also tweak the alignment a bit.

Here's FreeMonoBold, with a bit larger size. It'll do for a while. It's not that great, so I think we will want to get some handling for the ingame font. But that can wait now that we have the actual text display working nicely.



Now on to identifying items. Stage 1 is done, so stage 2 is next.

But it looks like we hit a snag on level 2. There is a centipede monster in this stage, which is stored as a series of segments in the graphics file, so we can't just draw a single sprite. We need a method for creating a compound image to draw him properly. Also, sprite 73 appears to be a hidden pickup. We will want to indicate these some way (i.e. by making them semi-transparent, probably). I will tackle all this tomorrow. Both cases should be just a matter of pre-processing the images before creating the corresponding sprite object and shouldn't require their own types of sprites.

day10.zip is available.

Hello folks. Let's get things restructured to allow item varients. I'm also going to introduce two new classes to make things more flexable: an object record class so I don't need to manually specify decoding every time, and a sprite class. The sprite class will be used to allow custom handling for certain sprites, like the player sprite (which needs an offset).

The objrecord class gets added to the xargonmap file:

Code Select Expand


class objrecord(object):
    def __init__(self, record):
        self.rawdata = record
        (self.sprtype, self.x, self.y) = record[0:3]
        (self.width, self.height, self.subtype) = record[5:8]


With minor changes to init and debug output methods to compensate.

Next we need to update our sprite db and graphics file (i.e. debug image) to support the two identifiers. Let's do the debug image first. Note that I'm changing this to draw a slightly larger image if needed, and make the image rectangle semi-transparent. I also stopped bothering to pick different shades of gray.

Code Select Expand


@staticmethod
def debugimage(index, subindex, width, height):
    """ Creates a debug image for sprites """
    if width > 0 and height > 0:
        # Provide sufficient space to display text
        imgwidth = max(width, 32)
        imgheight = max(height, 16)
        tempimage = Image.new("RGBA", (imgwidth, imgheight))
        pen = ImageDraw.Draw(tempimage)
        pen.rectangle(((0, 0), (width, height)), fill=(64, 64, 64, 128))
        pen.text((imgwidth/2 - 15, imgheight/2 - 6), '{}:{}'.format(index,subindex),
            font=imagefile.debugfont, fill=(255,255,255))
        return tempimage
    else:
        # 1 pixel transparent image
        return Image.new("RGBA", (width, height))


And the spritedb changes for both the two identifiers, and the dedicated sprite class. I also get rid of the two record structures, since that appears to be (another) misinterpretation.

Code Select Expand


class spritedb(object):
    def addsprite(self, sprtype, subtype, sprite):
        if sprtype not in self.sprites:
            self.sprites[sprtype] = {}
        self.sprites[sprtype][subtype] = sprite

    def __init__(self, graphics):
        self.sprites = {}
        self.mapsprites = {}

        # Manually-defined sprites (i.e. special handling needed
        self.addsprite(0, 4, sprite(graphics.records[6].images[9], yoffs=-8))

        # Simple sprite mapping. Stage sprites, then Map sprites
        for (sprtype, subtype, recnum, imagenum) in [(25, 0, 35, 2), # Monsters
                (33, 5, 37, 5), # Pickups
                (51, 0, 36, 33), # Clouds
                (5, 0, 47, 8)]: # Map Player
            self.addsprite(sprtype, subtype, sprite(graphics.records[recnum].images[imagenum]))

        # Empty sprites:
        # For future reference, possible meanings are:
        # 17-1: Respawn point
        # 63-3: Start??
        for sprtype, subtype in [(17,1), (63,3), (19,0)]:
            self.addsprite(sprtype, subtype, sprite(graphics.debugimage(sprtype, subtype, 0, 0)))

        # Cache a reference to the graphics object for future use
        self.graphics = graphics

    def drawsprite(self, mappicture, objrec):
        if objrec.sprtype not in self.sprites or \
                objrec.subtype not in self.sprites[objrec.sprtype]:
            self.addsprite(objrec.sprtype, objrec.subtype, sprite(
                self.graphics.debugimage(objrec.sprtype, objrec.subtype,
                objrec.width, objrec.height)))

        self.sprites[objrec.sprtype][objrec.subtype].draw(mappicture, objrec)

class sprite(object):
    def __init__(self, image, xoffs=0, yoffs=0):
        self.image = image
        self.xoffs = xoffs
        self.yoffs = yoffs

    def draw(self, mappicture, objrec):
        # When pasting masked images, need to specify the mask for the paste.
        # RGBA images can be used as their own masks.
        mappicture.paste(self.image, (objrec.x +self.xoffs,
            objrec.y +self.yoffs), self.image)




Excellant. Now it's time to ACTUALLY get decoding things, since we won't end up with overlapping definitions any more. Things a pretty straightforward at this point. Here's the properly identified first section of the map:



And a bit further into level 1:



But it looks like our interior room has a special sprite for the in-game text.



That's going to require something special to implement. Also, we're going to want to indicate what is INSIDE a present, since we can decode that information. That too, will require special handling. With the architecture we have now, we can simply set up subclasses of the sprite class to draw more complicated information. Unfortunately, we're out of time for today, so that will have to be tomorrow.

day9.zip is available.

Looking at the decoding from yesterday again, I think I misaligned the record start after all. I think the 20 00 at the start of the record area is actually the X coordinate for the first record. That means that the trailing number of 05 is its own thing. Let's move that around and see if that fixes our "almost correct" alignment for the pickups at the start. Remember, we expected either
0-based:
(3, 11) (3, 12) (2, 13) (4, 13)

or 1-based:
(4, 12) (4, 13) (3, 14) (5, 14)

And we get:

Code Select Expand


32  208 0   0   16  16  5   0   0   0   0   0   0   0   0   0   33  2   13
48  176 0   0   16  16  5   0   0   0   0   0   0   0   0   0   33  3   11
64  208 0   0   16  16  5   0   0   0   0   0   0   0   0   0   33  4   13
48  192 0   0   16  16  5   0   0   0   0   0   0   0   0   0   59  3   12


Coordinates match up, but the item ID is off for the last one. Looks like we are off by TWO fields. Let's fix that again and get:

Code Select Expand


33  32  208 0   0   16  16  5   0   0   0   0   0   0   0   0   0   2   13
33  48  176 0   0   16  16  5   0   0   0   0   0   0   0   0   0   3   11
33  64  208 0   0   16  16  5   0   0   0   0   0   0   0   0   0   4   13
33  48  192 0   0   16  16  5   0   0   0   0   0   0   0   0   0   3   12


I also switched the half-words to signed due to the strange 65535 entries. Nothing appears to "normally" approach the 16 bit boundary, so this should be safe. I also changed how I skip/capture the apparently unused map region. Hopefully I can find a pattern for the maps with different background colours, when I get there. Here's the updated code:

Code Select Expand


# Decode the object header then the object list
objrecord = '<B12h2B2H'

(numobjs,) = struct.unpack('<H', mapfile.read(2) )

self.objs = [struct.unpack(objrecord,
    mapfile.read(struct.calcsize(objrecord)) )
    for i in range(numobjs)]

# There always appears to be a 0x61 byte unknown region between
# the records and strings. Let's just collect it as bytes for now.
unknownregion = '<97B'
self.unknown = struct.unpack(unknownregion,
    mapfile.read(struct.calcsize(unknownregion)) )


Okay, now we think we have a reasonable decoding, and a few sample mappings (corrected due to the misalignment):
0 - Player start
33 - Lolly Pop
25 - Monster

Unfortunately, pickup/monster/object sprites are typically defined in code due the additional logic that is typically needed. It's unlikely we will find a direct mapping like we did for the Tiles. That said, there usually isn't an overwhelming number of interactable objects in a game, so we should be able to identify them as-we-go. We know there aren't more than 256 of them, after all. In order to track the mappings, we should start a new Python file for the sprite database. This will contain a lookup table for record entries, and a method to grab the correct sprite for a given sprite ID. We will also re-use the debug code from unknown tiles to provide placeholders for unknown sprites.

Finally, we need to make a decision on how sprites get drawn into the world. There are two options: have the mapper handle everything and add additional logic there, or defer processing to the sprite file, and pass in a reference to the map-in-progress. I'm going to choose the later, because it helps keep all the sprite processing in one place. Here's the first cut at the sprite file:

Code Select Expand


lookup = {  0: (6, 9),
           25: (35, 2),
           33: (37, 5)
         }

def drawsprite(mappicture, graphics, spriterecord):
    (spriteid, x, y) = spriterecord[0:3]
    (width, height) = spriterecord[5:7]

    if spriteid in lookup:
        (recnum, imagenum) = lookup[spriteid]
        spriteimage = graphics.records[recnum].images[imagenum]
    else:
        spriteimage = graphics.unknown[spriteid]

    # When pasting masked images, need to specify the mask for the paste.
    # RGBA images can be used as their own masks.
    mappicture.paste(spriteimage, (x, y), spriteimage)


And the addition to the mapper:

Code Select Expand


for objrecord in mapdata.objs:
    spritedb.drawsprite(self.mappicture, graphics, objrecord)




Looking good. A couple things to note that we will need to take care of:
1) There are some objects that appear functional but not visual in nature (object IDs 17 and 63). We could provide markup to indicate what we think they mean, but I think a better approach would be to simply not draw anything.
2) The player start appears slightly in the ground. We will either need to figure out an alignment algorithm to correct this sort of thing, or make a special case for the player sprite.
3) The sprites have Dimensions. We should take this into account for our debug images to make them easier to identify by shape as well as location.

To make this sort of thing easier, I'm going to expand our spritedb file into an actual class which will keep a copy of the corresponding sprite images. This way it can generate debug images on-the-fly when a sprite is not found, and we can make specific entries for empty sprites. To consolidate functionality, I made the debug image function return empty sprites if called with 0 width and height.

Code Select Expand


class spritedb(object):
    def __init__(self, graphics):
        self.sprites = {}

        # Simple sprite mapping:
        for (spriteid, recnum, imagenum) in [(0, 6, 9), (25, 35, 2),
                (33, 37, 5)]:
            self.sprites[spriteid] = graphics.records[recnum].images[imagenum]

        # Empty sprites:
        # For future reference, possible meanings are:
        # 17: Respawn point
        # 63: Start??
        for spriteid in [17, 63]:
            self.sprites[spriteid] = graphics.debugimage(spriteid, 0, 0)

        # Cache a reference to the graphics object for future use
        self.graphics = graphics

    def drawsprite(self, mappicture, spriterecord):
        (spriteid, x, y) = spriterecord[0:3]
        (width, height) = spriterecord[5:7]

        if spriteid not in self.sprites:
            self.sprites[spriteid] = self.graphics.debugimage(spriteid, width, height)

        # When pasting masked images, need to specify the mask for the paste.
        # RGBA images can be used as their own masks.
        mappicture.paste(self.sprites[spriteid], (x, y), self.sprites[spriteid])


Updated debug image function:

Code Select Expand


def debugimage(index, width, height):
    """ Creates a debug image for sprites """
    if width > 0 and height > 0:
        colour = (index, index, index)
        tempimage = Image.new("RGBA", (width, height), colour)
        textcolor = (255, 255, 255) if index < 96 else (0, 0, 0)
        pen = ImageDraw.Draw(tempimage)
        pen.text((width/2 - 7, height/2 - 6), '{:02}'.format(index),
            font=imagefile.debugfont, fill=textcolor)
        return tempimage
    else:
        # 1 pixel transparent image
        return Image.new("RGBA", (width, height))




And now, we get to identifying. Time to fire up DosBox. Those 51 sprites look like movable clouds, so I'll go find their sprite first. Once I get in-game, I'm going to go ahead and start identifying world map sprites too.

Hrm, it actually looks like the map goes by different sprite IDs!



That's fine, we can maintain two lookups. I'm going to go ahead and split it, then identify a few sprites. Then again, that doesn't look to be enough info. For instance, the 88 sprites on the map screen appear to be mountains, but mountains have two sides. I think there is some sort of sub-ID going on here. And I think I found it:

Code Select Expand


88  288     112 0   0   48  53  2   0   0   0   0   0   0   0   0   0
88  336     112 0   0   48  53  3   0   0   0   0   0   0   0   0   0
88  1232    592 0   0   48  53  6   0   0   0   0   0   0   0   0   0
88  768     112 0   0   48  53  2   0   0   0   0   0   0   0   0   0
88  816     112 0   0   48  53  3   0   0   0   0   0   0   0   0   0
88  976     176 0   0   48  53  0   0   0   0   0   0   0   0   0   0
88  1024    176 0   0   48  53  1   0   0   0   0   0   0   0   0   0


I think it's that column that counts 2, 3, 6, 2, 3, 0, 1. Next I'm going to need to expand the sprite identification to take that into account. But that's a task for tomorrow. I'll also need to figure out some way to clearly indicate the sub ID in the debug image. I may need to move back to 8 point font

day8.zip is available for anyone who wants it.

Hello and welcome back. Today we will be looking at the rest of the map format. We're looking for two major things to include in our map:
1) some sort of header information that will hopefully give us more information on background colours
2) Monsters and Pickups!

Remember the garbage at the bottom of this image?


Well, let's try to figure out what it means. There's obviously a repeating pattern here, but it's a bit skewed because we were decoding two bytes at a time. From the image, it looks about 15-16 pixels, or probably 31 bytes in length. Let's see if we can find the bounds of this record in a hex editor. The map is 128*64*2 bytes long, so we should start at offset 0x4000.



Looks about right. But now we need to find the bounds of this region, and to figure out what comes before and after it. Let's start at the end of the file instead.



Well, that's very clearly delineated. There is a number followed by a string then the next string, etc. The number appears to be the string length. The previous section also has a clear empty buffer region between, so I'm going to assume that the last record ends with a non-zero number. If I then grab the last record, it looks like:
F0 02 00 00 00 00 10 00 0C 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 05

That seems like a fairly likely record alignment. The first two characters correspond to a 16 number of 752, which is a reasonable decoding. The F0 is at offset 0x4A8E, so let's keep going back in intervals of 31 until we reach a reasonable first record. (0x4000 - 0x4A8E) / 0x57.29, so it looks like we have approximately 0x57 (87) records here. My first guess places the start of these records at address 0x4005.

Lining up the first few records:

Code Select Expand


80 00 00 00 00 00 18 00 28 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 20 00
90 00 00 00 00 00 10 00 10 00 01 00 00 00 00 00 01 00 00 00 04 00 A6 48 00 00 00 00 3E D0 02
E0 00 00 00 00 00 10 00 10 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 3D D0 02
F0 00 00 00 00 00 10 00 10 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 3D 80 06
00 03 00 00 00 00 10 00 10 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 3E 80 06
F0 02 00 00 00 00 10 00 10 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 3F 20 00


Looks right to me. That means we have 5 unique bytes:
58 00 00 20 00

0x58 = 88, which appears to be the number of records we have. That leaves us with 0x20 (i.e. 32). Let's look at the headers for a couple different stages set together:

Code Select Expand


58 00 00 20 00
58 00 00 20 00
6C 00 00 30 00
AD 00 00 B0 03
B6 00 00 30 00
85 00 00 B0 01
A1 00 00 90 00
B6 00 00 E0 01
9A 00 00 40 07
A1 00 00 90 00
B5 00 00 80 00
2E 00 00 30 07
70 00 00 30 02


Well, I can't figure out a correlation right now, but I think this is a 16 bit number.

Let's assign bit decoding characters for the repeating record. I'm going to grab a few more examples just to get a clearer picture. This may be incorrect, but we need to start with something:

Code Select Expand


H     H     H     H     H     H     H     H     H     H     H     B  B  H     H     B  H
80 00 00 00 00 00 18 00 28 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 20 00
90 00 00 00 00 00 10 00 10 00 01 00 00 00 00 00 01 00 00 00 04 00 A6 48 00 00 00 00 3E D0 02
E0 00 00 00 00 00 10 00 10 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 3D D0 02
F0 00 00 00 00 00 10 00 10 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 3D 80 06
00 03 00 00 00 00 10 00 10 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 3E 80 06
F0 02 00 00 00 00 10 00 10 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 3F 20 00
A0 00 01 00 00 00 00 00 00 00 03 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 48 90 07
B0 00 00 00 00 00 10 00 10 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 11 A0 07
B0 00 00 00 00 00 10 00 10 00 00 00 00 00 00 00 00 00 00 00 04 00 A5 48 00 00 00 00 48 A0 07
B0 00 00 00 00 00 10 00 10 00 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 21 40 07
A0 01 00 00 00 00 10 00 10 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 21 50 07
90 01 00 00 00 00 10 00 10 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 21 60 07


Or, condensed, '<11H2B2HBH'. Time to update our map script to decode this information and write it out to a debug CSV so we can hopefully figure something useful out of it (like an object ID and coordinates).

Code Select Expand


class xargonmap(object):
    def __init__(self, filename):
        # Grab the map from the file name (sans ext)
        (temppath, tempfname) = os.path.split(filename)
        (self.name, tempext) = os.path.splitext(tempfname)

        # Load the map data as a 64*128 array of 16 bit values:
        mapfile = open(filename, 'rb')
        pattern = '<{}H'.format(64*128)

        self.tiles = struct.unpack(pattern,
            mapfile.read(struct.calcsize(pattern)) )

        # Decode the object header then the object list
        objheader = '<HBH'
        objrecord = '<11H2B2HBH'

        (numobjs, blank, self.unknown) = struct.unpack(objheader,
            mapfile.read(struct.calcsize(objheader)) )

        self.objs = [struct.unpack(objrecord,
            mapfile.read(struct.calcsize(objrecord)) )
            for i in range(numobjs)]

        # There always appears to be a 0x5E spacer region
        mapfile.read(0x5E)

        # Capture any strings until the end of the file
        self.strings = []
        sizebytes = mapfile.read(2)
        while (len(sizebytes) == 2):
            (stringlen,) = struct.unpack('<H', sizebytes)
            self.strings.append(mapfile.read(stringlen))
            mapfile.read(1)
            sizebytes = mapfile.read(2)

        mapfile.close()


    def debugcsv(self):
        # Remember that the map is height-first. We need to convert to
        # width-first. This only outputs tile data for now.
        with open(self.name + '.csv', 'wb') as csvfile:
            writer = csv.writer(csvfile)
            for y in range(64):
                writer.writerow([self.tiles[x*64+y] for x in range(128)])

        # Next, output the object list:
        with open(self.name + '_objs.csv', 'wb') as csvfile:
            writer = csv.writer(csvfile)
            writer.writerows(self.objs)

        # Finally, the header and strings list:
        with open(self.name + '_info.csv', 'wb') as csvfile:
            writer = csv.writer(csvfile)
            writer.writerow([self.unknown] + self.strings)


And a subset of the object csv output:

Code Select Expand


176 0   0       16  16  4   0   0   0   0   0   0   0   0   0   17  1952
176 0   0       16  16  0   0   0   0   0   4   165 72  0   0   72  1952
176 0   0       16  16  4   0   0   0   0   0   0   0   0   0   33  1856
416 0   0       16  16  2   0   0   0   0   0   0   0   0   0   33  1872
400 0   0       16  16  2   0   0   0   0   0   0   0   0   0   33  1888
400 0   0       16  16  2   0   0   0   0   0   0   0   0   0   33  1904
416 0   0       16  16  2   0   0   0   0   0   0   0   0   0   21  1936
240 0   0       16  16  0   0   0   0   0   0   0   0   0   0   7   1584
788 9   65535   184 8   0   0   0   0   0   4   55  68  0   0   22  1584
752 0   0       16  12  0   0   0   0   0   0   0   0   0   0   22  1600
736 0   0       16  12  0   0   0   0   0   0   0   0   0   0   22  1600
704 0   0       16  12  0   0   0   0   0   0   0   0   0   0   22  1584
720 0   0       16  12  0   0   0   0   0   0   0   0   0   0   22  1744
704 0   0       16  12  0   0   0   0   0   0   0   0   0   0   22  1760
720 0   0       16  12  0   0   0   0   0   0   0   0   0   0   22  1744
736 0   0       16  12  0   0   0   0   0   0   0   0   0   0   22  1760
752 0   0       16  12  0   0   0   0   0   0   0   0   0   0   21  1440
464 0   0       16  16  0   0   0   0   0   0   0   0   0   0   72  1936
160 0   0       32  16  11  0   0   0   0   0   0   0   0   0   51  160
128 1   0       80  16  0   0   0   0   0   0   0   0   0   0   51  1824


Well, whatever is in that 65535's column, it is probably signed. However, it's unlikely that data will be relevant for us (since it's normally zero). We need to pick out likely coordinate and item ID columns. The easiest way to do this is to locate a known object (or pattern of objects) in the game world and try to find their representation.

This classic screenshot provides some useful objects. All of the same type in close proximity:


Opening up the pixelmap image in GIMP, drawing in the item locations, then checking their coordinates results in the following zero-based indicies:
(3, 11) (3, 12) (2, 13) (4, 13)

1-based:
(4, 12) (4, 13) (3, 14) (5, 14)

And I'm coming up pretty empty. I'm not even sure what could look like coordinates or an index in the data I have. Nothing is obviously a coordinate system, and a direct index would require a number between 0 and 8192, and nothing goes anywhere near this value. But I have a strange idea that the indicies could possibly be given in absolute pixel coordinates. Let's divide the first and last columns by 16 and see if we see any patterns.

And.... (drum roll):

Code Select Expand


176 0   0   16  16  0   0   0   0   0   0   0   0   0   0   33  32 | 11  2
208 0   0   16  16  5   0   0   0   0   0   0   0   0   0   33  48 | 13  3
176 0   0   16  16  5   0   0   0   0   0   0   0   0   0   33  64 | 11  4
208 0   0   16  16  5   0   0   0   0   0   0   0   0   0   33  48 | 13  3


Hrm, not a perfect match, but close. Also, every entry appears to be within the bounds for a 64 x 128 map, and appears sane. I'm not going to implement today, but I have a theoretical decoding. I will number the columns below:

Code Select Expand


1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17
176 0   0   16  16  0   0   0   0   0   0   0   0   0   0   33  32 | 11  2
208 0   0   16  16  5   0   0   0   0   0   0   0   0   0   33  48 | 13  3
176 0   0   16  16  5   0   0   0   0   0   0   0   0   0   33  64 | 11  4
208 0   0   16  16  5   0   0   0   0   0   0   0   0   0   33  48 | 13  3


Column 1: Y coordinate * 16
Column 17: X coordinate * 16
Columns 4 and 5: Nominal sprite dimensions
Column 16: Sprite type

And a few tenative identifications:
17 : Player Start
25: Wandering Monster thing
33 : Lolly Pop pickup

The nominal sprite dimensions entry I guessed because I noticed the coordinates 2, 8 would put something pretty close to the start of the level and the dimensions were 24, 40 (same as the player sprite). Also, the first-level monster sprite dimensions are 38, 25, which shows up several times in this file.

day7.zip is available for anyone who wants it.

Thanks for the feedback guys. To expand on Darkwolf's comment, Xargon is a very simple game with some basic data structures. It makes it a good example, but note that other games may be more difficult. Both games I did prior to this (ROTT and ShadowCaster) had more complicated sprite schemes. ROTT used an RLE encoding scheme as DarkWolf described, while ShadowCaster actually had a simple scheme of specifying the start and end pixel in a column, but not otherwise compressing the image. That said, a lot of games have had some prior investigation, or even a source code release, which can help reduce the guesswork immensely. As I mentioned, the source code for Xargon has been released, but I'm intentionally avoiding it to make this guide (hopefully) more useful.

To start off today's exercise, I'm going to play around in a spreadsheet. Exciting stuff. Basically, for every known sprite thus far, I want to figure out the exact relationship between the TILES file, the Record ID, and the Map ID. One thing I noticed yesterday was that the Record ID column in the TILES file appeared to shift by a 256 boundary for each record. We should be able to use the classic divide and modulus pair to split out the record number. I just need to confirm to make sure we don't start drifting for whatever offset I end up calculating.

Records appear to be easy. The record number is just the TILES entry / 256 - 64, and the entry in the record is the TILES entry % 256 -1. The only two things that we need to pay special attention to are:
1) The "extra" tile we mentioned before (i.e. that I thought might be garbage data) appears to be referred to by position -1. That said, index -1 in Python already refers to the last element in a list, so we don't actually need to handle this explicitly.
2) The tiles file refers to record 49, which does not exist. We will just need to keep an eye out if this happens. I expect this is just placeholder data (maybe for the registered version).

The Tile IDs appear to just be offset by C000, which is pretty simple. I'll need to figure out the tiles that are in the 0 to FF range on the map, but that can wait. I'll just put an IF statement to avoid processing them for now.

So let's expand our xargontiles.py file to do the lookup for us. We have two options for linking together the graphics and tile files:
1) Pass in the Graphics file when creating the Tiles file and store a reference for future use
2) Pass in the Graphics file for the lookup operation to the Tiles file

Since I prefer to keep each class tailored to just the file it is responsible for, I will be doing the second option. The xargonmapper.py is the only file that will know about all the related files and will tie them together.

First, we need to set up xargontile to do easier lookups. I will expand the initialization routine to populate a dictionary of tile to record mappings:

Code Select Expand


def __init__(self, filename):
    filesize = os.path.getsize(filename)
    infile = open(filename, 'rb')

    self.tiles = []
    self.lookup = {}

    commonheader = '<3HB'
    while infile.tell() < filesize:
        headerdata = struct.unpack(commonheader,
            infile.read(struct.calcsize(commonheader)) )
        stringlen = headerdata[3]
        self.tiles.append(headerdata[0:3] +
            struct.unpack('<{}s'.format(stringlen), infile.read(stringlen)) )

        self.lookup[headerdata[0]] = headerdata[1]


Then we write the lookup routine itself to use this lookup and take the offsets into account:

Code Select Expand


def gettile(self, graphics, tilenum):
    if tilenum < 0xC000:
        return graphics.tilelookup[tilenum]
    else:
        graphindex = self.lookup[tilenum - 0xC000]

        recnum = graphindex / 256 - 64
        recindex = graphindex % 256 - 1

        return graphics.records[recnum].images[recindex]


And this results in a very minor change to xargonmapper:

Code Select Expand


def __init__(self, graphics, tiledata, mapdata):
    self.mappicture = Image.new("RGB", (128*16, 64*16) )
    self.name = mapdata.name

    for index, tileval in enumerate(mapdata.tiles):
        # Remember: maps are height first
        (x, y) = (index/64, index%64)
        self.mappicture.paste(tiledata.gettile(graphics, tileval),
            (x*16, y*16) )


Then we run it!



And would you look at that? THAT is progress. But it's not perfect, and it has some glitches we need to investigate. Notably, we haven't looked into the map index 0 -> FF area yet. I also noticed a couple glitches over in this part of the map which we need to investigate further:



But first, the region that doesn't start with C000:



And let's start identifying some tiles. I already have a theory that these are simply direct indicies into the TILES file, so let's test that out with the black tiles (00AD it looks like). 0xAD -> 173 decimal, which has the name of 0NO and record 9, index 16. Sure enough, that tile is totally black. Let's implement it and see how it turns out:

Code Select Expand


def gettile(self, graphics, tilenum):
    if tilenum < 0xC000:
        graphindex = self.lookup[tilenum]
    else:
        graphindex = self.lookup[tilenum - 0xC000]

    recnum = graphindex / 256 - 64
    recindex = graphindex % 256 - 1

    return graphics.records[recnum].images[recindex]




Almost there, but we ran into the same glitch again. Let's look into that. I'm just going to open up the debug CSV to grab the tile value that screwed up... and it looks like 174. In the TILES array, 174 is called RNSLDGM... and it's in record 10, index -1. ARGH. Looks like the out-of-place "extra" graphics really are out of place! And I can't figure out a clear reason why the "extra" tile looks wrong. However, this tile in particular looks like record 10, index 3 (although that is also assigned a tile name).

What I'm going to is remove the debug image list from before, since we have the tiles nominally identified. Instead, I will create a much shorter list to handle the cases where we run into a -1 tile and re-direct it to the correct index (if, in fact, such a tile exists). Hopefully we can get something working.

Code Select Expand


neg1mapping = {8: (8, 24), 10: (10, 3)}

def gettile(self, graphics, tilenum):
    if tilenum < 0xC000:
        graphindex = self.lookup[tilenum]
    else:
        graphindex = self.lookup[tilenum - 0xC000]

    recnum = graphindex / 256 - 64
    recindex = graphindex % 256 - 1

    # Negative 1 index needs special handling. The correct image does
    # not always appear in the correct spot.
    if recindex == -1:
        if recnum in self.neg1mapping:
            (recnum, recindex) = self.neg1mapping[recnum]
        else:
            return graphics.unknown[recnum]

    return graphics.records[recnum].images[recindex]


And a modification to the debug tile code:

Code Select Expand


@staticmethod
def debugimage(index):
    """ Creates a 16x16 debug image for tiles """
    colour = (index, index, index)
    tempimage = Image.new("RGBA", (16, 16), colour)
    textcolor = (255, 255, 255) if index < 96 else (0, 0, 0)
    pen = ImageDraw.Draw(tempimage)
    pen.text((1, 2), '{:02}'.format(index), font=imagefile.debugfont, fill=textcolor)
    return tempimage


So yes, it looks like all the glitched tiles ARE the -1 tiles. Unfortunately, the tile we picked was wrong; the correct tile should have more of a highlight. We also need to find a suitable rope tile, it looks like.



And I can't find either of them! *cries*.

Wait a minute. I am an idiot. Raise your hand if you see the bug below:

Code Select Expand


for recnum, record in enumerate(self.records):
    if recnum == 53:
        record.loadimages(palette2, skipimages=1)
    elif recnum == 5:
        record.loadimages(palette1, skipimages=1)
    else:
        record.loadimages(palette1, skipimages=1)


That's right, I ALWAYS skip the first image! GAH! This explains the whole subtract 1 thing (which, obviously now, is wrong). Let's fix both problems and re-run to generate:



Yay! For run, I also ran it on the rest of the maps, and most everything looks okay, except for a few maps:



Looks like there may be some sort of transparency behaviour with the sky colour on other maps? Or a different colour palette? I'll need to investigate when I decode more of the map format. That will be tomorrow's topic.

And finally, day6.zip is available for anyone who wants it. It includes the full output images thus far if anyone wants to see what the current versions of the rest of the maps look like.

Yeah, I'm thinking the black box method is only really practical with old DOS games. The newer games are more likely to be using strange compression schemes and alternate image formats. That said, some games may just use standard image formats (like PNG), so it may be entirely possible to start looking for headers of known image files.

Then again, the entire resource file is probably compressed too.

On to Day 5: the day where I stop teasing and actually create a map.

To support this, we need to start a lookup routine in the graphics file to get the corresponding tile for a map tile index. Unfortunately, we don't know the mapping yet, so we need to pre-populate with debug tiles. Since we found out that the tile index is only an 8-bit number, we should be able to fit it (in hex notation) in the 16 x 16 pixel debug tile. This may make the CSV we made obsolete, but what can you do.

We need a font to generate the text, so I'm going to copy in the Android favourite, DroidSans. Here's the routine to create a simple debug tile (adapted from my Shadow Caster mapper) and the font declaration:

Code Select Expand


debugfont = ImageFont.truetype("DroidSans.ttf", 12)

@staticmethod
def debugimage(colour, text):
    """ Creates a 16x16 debug image for tiles:

    colour -- the background colour for the image.
    text -- the text to display
    """
    tempimage = Image.new("RGBA", (16, 16), colour)
    textcolor = (255, 255, 255) if colour[0] < 96 else (0, 0, 0)
    pen = ImageDraw.Draw(tempimage)
    pen.text((1, 2), text, font=imagefile.debugfont, fill=textcolor)
    return tempimage


Here's the addition to the __init__ method for the graphics file:

Code Select Expand


# Group by map tile:
self.tilelookup = [self.debugimage( (i, i, i), '{:02X}'.format(i) )
    for i in range(256) ]


And the lookup method, which will change once we figure out real mappings:

Code Select Expand


def gettile(self, tilenum):
    return self.tilelookup(tilenum)


Now, we start a new file for generating the map. In this file, we're going to import the other two files we wrote in order to use their features in tandem. This will start out extremely simple: we will create a large Image object to hold the map, then paste every debug tile according to its coordinates. Since the map is a linear set of tiles, we need to use integer division and modulus (i.e. the "remainder") to split that up into X, Y coordinates.

Here's the initial mapper file. I set up the mapper as a class, but you can do it as a standalone set of functions as well. Personally, I like to split it up into phases. the Shadow Caster had initialize, generate and save phases, but I think we can combine the first two in this case.

Code Select Expand


import sys
from PIL import Image
from xargonmap import xargonmap
from xargongraphics import imagefile

class xargonmapper(object):
    def __init__(self, graphics, mapdata):
        self.mappicture = Image.new("RGB", (128*16, 64*16) )
        self.name = mapdata.name

        for index, tileval in enumerate(mapdata.tiles):
            # Remember: maps are height first
            (x, y) = (index/64, index%64)
            self.mappicture.paste(graphics.gettile(tileval),
                (x*16, y*16) )

    def save(self):
        self.mappicture.save(self.name + '.png')


if __name__ == "__main__":
    if len(sys.argv) < 3:
        print """Usage: python xargonmapper.py [Graphics File] [Map File(s)...]
TODO
"""
    else:
        xargonimages = imagefile(sys.argv[1])
        for filename in sys.argv[2:]:
            themap = xargonmap(filename)

            print "Generating Map '{}'".format(themap.name)
            mapper = xargonmapper(xargonimages, themap)
            print "Saving Map '{}'".format(themap.name)
            mapper.save()


And here's our first output with no tiles identified:



Remember this screenshot from Day 2?



Well, that's same area. So now we need to go look through our extracted tiles and match a few to IDs. Usually we can figure out a trend to unlock the rest for us; otherwise it can become a bit tedious. I see a few tiles in graphics record #8, so I'm going to go ahead and map those. Actually, I already see a pattern: it looks like map tile values 1 to at least 0A correspond to the first few tiles in group 8. But there appears to be a discontinuity at index 6. GRR. Let's do what we can with it.

Code Select Expand


    def gettile(self, tilenum):
        if tilenum >= 2 and tilenum <=5:
            return self.records[8].images[tilenum-2]
        elif tilenum >= 7 and tilenum <=10:
            return self.records[8].images[tilenum-3]
        else:
            return self.tilelookup[tilenum]


Not too bad so far, but I can see this getting awkward after a bit. Let's see what this does to the map:



Okay, that's progress. Now let me go ahead and try to identify some of the other tiles... gah, what is that??



Looks like I was wrong about the 8-bit tiles. This all falls under the red area in the original visualisations. Looking a bit closer at my grayscale meta files in GIMP show that even the non-black regions appear to have a few very-close-but-not-identical colour values. Time so switch back to 16-bit tiles (grumble). I'm not quite sure how to make 4-digits visible on a 16 pixel tile though. I'll try, but I may need to rely more on the CSV.

Code Select Expand


debugfont = ImageFont.truetype("DroidSans.ttf", 8)

@staticmethod
def debugimage(index):
    """ Creates a 16x16 debug image for tiles """
    colour = (index%256, index/256, 0)
    tempimage = Image.new("RGBA", (16, 16), colour)
    textcolor = (255, 255, 255) if colour[0] < 96 and colour[1] < 96 else (0, 0, 0)
    pen = ImageDraw.Draw(tempimage)
    pen.text((4, 0), '{:02X}'.format(index/256), font=imagefile.debugfont, fill=textcolor)
    pen.text((4, 8), '{:02X}'.format(index%256), font=imagefile.debugfont, fill=textcolor)
    return tempimage


8-point font FTW. Just readable. It takes a noticeable delay to generate 65535 debug tiles though . Now to adapt my previous identifications (+ a couple more):

Code Select Expand


def gettile(self, tilenum):
    if tilenum == 0xC000:
        return self.records[9].images[15]
    elif tilenum == 0xC001:
        return self.records[8].images[24]
    elif tilenum >= 0xC002 and tilenum <=0xC005:
        return self.records[8].images[tilenum-0xC002]
    elif tilenum >= 0xC007 and tilenum <=0xC00A:
        return self.records[8].images[tilenum-0xC003]
    elif tilenum >= 0xC1AF and tilenum <=0xC1B3:
        return self.records[14].images[tilenum-0xC1AF+18]
    elif tilenum >= 0xC0F5 and tilenum <=0xC0F6:
        return self.records[25].images[tilenum-0xC0F5+2]
    elif tilenum >= 0xC0D3 and tilenum <=0xC0D6:
        return self.records[25].images[tilenum-0xC0D3+9]
    elif tilenum >= 0xC2DE and tilenum <=0xC2DF:
        return self.records[25].images[tilenum-0xC2DE+13]
    elif tilenum >= 0xC2E2 and tilenum <=0xC2E3:
        return self.records[25].images[tilenum-0xC2E2+19]
    elif tilenum >= 0xC2EE and tilenum <=0xC2F1:
        return self.records[25].images[tilenum-0xC2EE+31]
    else:
        return self.tilelookup[tilenum]




No more misidentified tiles, but we aren't exactly coming up with much of a trend, even if we were to calculate out the offsets (i.e. -0xC0F5+2 = -0xC0F3, but -0xC0D3+9 = -0xC0CA). It seems that almost every group of tiles, even those in the same record, ends up discontinuous. At this point, we have two options. Either keep identifying tiles manually (and up with some sort of mapping database), or see if there is some information we are missing. Hey now, there's a file called TILES.XR1. Let's see what's in it. Looks like a whole bunch of strings with some supplementary information. It's a very regular file, so it should be fairly simple to decode.

Here's the last record in the file (with ASCII characters substituting the obvious text portion):
81 03 3A 5B 01 00 07 MPBRDG1

The last byte appears to be the string length, so it's just the first 6 to guess about. Since we are dealing with 16-bit tile values, I'm going to decode each of these at 16 bits on a hunch.

Here's our new file:

Code Select Expand


import struct, sys, os, pdb, csv
from PIL import Image, ImageFont, ImageDraw

class tilefile(object):
    debugfont = ImageFont.truetype("DroidSans.ttf", 8)

    def __init__(self, filename):
        filesize = os.path.getsize(filename)
        infile = open(filename, 'rb')

        self.tiles = []

        commonheader = '<3HB'
        while infile.tell() < filesize:
            headerdata = struct.unpack(commonheader,
                infile.read(struct.calcsize(commonheader)) )
            stringlen = headerdata[3]
            print headerdata
            self.tiles.append(headerdata[0:3] +
                struct.unpack('<{}s'.format(stringlen), infile.read(stringlen)) )

    def debug_csv(self, filename):
        with open(filename, 'wb') as csvfile:
            writer = csv.writer(csvfile)
            for recnum, tiledata in enumerate(self.tiles):
                writer.writerow((recnum,) + tiledata)


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargontiles.py [Tiles File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            xargontiles = tilefile(filename)
            xargontiles.debug_csv('tiles.csv')


And part of the output. First column is the index in the file, the rest is data from the file:

Code Select Expand


0   0   18704   1   0
1   1   18432   1   SK1
2   2   18433   1   SK2
3   3   18434   1   SK3
4   4   18435   1   SK4
5   5   18436   1   SK5
6   6   18707   1   CLD1
7   7   18437   1   VL
8   8   18438   1   VR
9   9   18439   0   S1L
10  10  18440   0   S1R
11  11  18708   1   CLD2
12  12  18709   1   CLD3
13  13  18441   1   R1
14  14  18442   1   R2
15  15  18443   1   R3
16  16  18444   1   R4
17  17  18445   3   GS1
18  18  18446   3   GS2
19  19  18447   1   R5
20  20  18448   1   R6
21  21  18449   0   S2
22  22  18450   0   S2C
23  23  18451   0   S2M
24  24  18452   0   S2B
25  25  18453   1   S2U
26  26  18454   1   S2BK
27  27  18710   1   CLD4
28  28  18711   1   CLD5
29  29  18712   1   CLD6


Still no direct correlation to the values we see on the map, as C000 (for example) is 49152 in decimal. Let me do one more thing today. I'm going to re-interpret some of our unknown GRAPHICS record header values as 16-bit numbers.

[code]
0   0   0
1   768     8591    128 1   2560    4608    8704    2   1   0
2   9359    5007    128 1   2048    3584    6656    2   1   0
3   14366   721     10  1   200     360     680     2   1   0
4   15087   6559    36  1   1744    3344    6544    8   4   0
5   21646   914     1   1   196     388     772     8   4   0
6   22560   32041   38  1   8122    16096   32032   8   0   0
7   54601   6880    27  1   1564    3020    5932    8   4   0
8   61481   6746    25  1   1700    3300    6500    8   4   0
9   68227   6746    25  1   1700    3300    6500    8   4   0
10  74973   12224   47  1   3196    6204    12220   8   4   0
11  87197   7523    28  1   1904    3696    7280    8   4   0
12  94720   3379    12  1   816     1584    3120    8   4   0
13  98099   3638    13  1   884     1716    3380    8   4   0
14  101737  22325   86  1   5848    11352   22360   8   4   0
15  124062  7264    27  1   1836    3564    7020    8   4   0
16  131326  8818    33  1   2244    4356    8580    8   4   0
17  140144  8041    30  1   2040    3960    7800    8   4   0
18  148185  7523    28  1   1904    3696    7280    8   4   0
19  155708  4674    17  1   1156    2244    4420    8   4   0
20  160382  3379    12  1   816     1584    3120    8   4   0
21  163761  7523    28  1   1904    3696    7280    8   4   0
22  171284  6746    25  1   1700    3300    6500    8   4   0
23  178030  19437   74  1   5032    9768    19240   8   4   0
24  197467  7264    27  1   1836    3564    7020    8   4   0
25  204731  9595    36  1   2448    4752    9360    8   4   0
26  214326  20991   80  1   5440    10560   20800   8   4   0
27  235317  18401   70  1   4760    9240    18200   8   4   0
28  253718  8559    32  1   2176    4224    8320    8   4   0
29  262277  4415    16  1   1088    2112    4160    8   4   0
30  266692  30196   77  1   7901    15900   30680   8   0   0
31  296888  8523    36  1   2128    5392    8080    8   0   0
32  305411  4308    7   1   1036    2044    4060    8   0   0
33  309719  11437   12  1   2840    5652    11216   8   0   0
34  321156  4775    8   1   992     1952    3872    8   0   0
35  325931  17409   18  1   4572    9072    18072   8   0   0
36  343340  10454   37  1   2650    5236    10156   8   0   0
37  353794  9499    36  1   2360    4688    9008    8   0   0
38  363293  11650   7   1   2880    6144    11436   8   0   0
39  374943  20246 

Before we start identifying tiles and creating the image map, there are two things I noticed yesterday about the image data that I want to investigate.

1) The GRAPHICS file table of contents had a few blank entries. My current algorithm skips those positions entirely. However, when we start identifying tile mappings, the fact that there was a field allocated for them may be important. I will therefore update my algorithm to specifically create blank places in order to keep the record alignment accurate. It also appears to have space for more records, so I'm going to at least process those regions in case there are more records in the registered version.

2) Record 4 and Record 49 have a garbage 64*12 image in each of them. Also, record 49 appears to have the wrong colour palette. The interesting thing about a 64*12 image is that it takes up 768 bytes. A full 256 colour palette also takes up 768 bytes, so I suspect these records are actually colour palettes. If so, we don't need our sample screenshot and we can pull the colour palette direct from the game data. It also means we can correct the palette for the Record 49 images, if this is accurate.

Also, as the source files are going to get a bit more plentiful (and of increasing length) I'm also going to change to only posting snippets of code and include the full listing in a zip at the end of this day's post.

Finally, because I'm paranoid, I'm going to put in a warning of a record still has data after we read the last image in that record. I *think* I'm decoding that right, but if there's more data, we don't want to miss it.


For the blank entries, it couldn't be easier. Increasing the size of the record removes the need for the seek command and the IF on the list comprehension:

Code Select Expand


header = '<128L'
headerdata = struct.unpack(header,
    graphicsfile.read(struct.calcsize(header)) )
header2 = '<128H'
headerdata2 = struct.unpack(header2,
    graphicsfile.read(struct.calcsize(header2)) )

# Create the image records using list comprehension
self.records = [imagerecord(graphicsfile, offset, size, palette)
    for (offset, size) in zip(headerdata, headerdata2)]


Instead, the IF needs to go inside the image record instead. It also needs to be added to the debug output so we don't get a ton of empty folders. I'll let the debug CSV actually attempt to write out data for these blank records so it's easier to see the breakdown. I'm also going to add my paranoid sanity check to the __init__ method now. Here is the updated imagerecord class:

Code Select Expand


class imagerecord(object):

    @staticmethod
    def maskimage(inimage):
        """ Masks colour 0 in the given image, turning those pixels
        transparent. Returns the resulting RGBA image.
        """
        tempmask = Image.new('L', inimage.size, 255)
        maskdata = list(tempmask.getdata())
        outimage = inimage.convert("RGBA")

        for pos, value in enumerate(inimage.getdata()):
            if value == 0:
                maskdata[pos] = 0

        tempmask.putdata(maskdata)
        outimage.putalpha(tempmask)
        return outimage

    def __init__(self, filedata, offset, size, palette):
        self.offset = offset
        self.size = size

        self.images = []

        if offset > 0:
            filedata.seek(offset)
            headerstruct = '<12B'
            self.header = struct.unpack(headerstruct,
                filedata.read(struct.calcsize(headerstruct)) )

            self.numimages = self.header[0]

            for tilenum in range(self.numimages):
                (width, height, unknown) = struct.unpack('<3B',
                    filedata.read(3))

                tile = Image.fromstring("P", (width, height),
                    filedata.read(width*height))
                tile.putpalette(palette)
                self.images.append(self.maskimage(tile))

            # Check to see if we actually loaded all data from this record
            leftover = self.offset + self.size - filedata.tell()
            if leftover > 0:
                print "Record at offset {} has {} bytes unaccounted for.".format(self.offset, leftover)
            elif leftover < 0:
                print "Record at offset {} read {} bytes beyond its boundary.".format(self.offset, -leftover)

        else:
            self.numimages = 0
            self.header = []


    def save(self, outpath):
        if self.numimages > 0:
            createpath(outpath)
            for tilenum, tile in enumerate(self.images):
                tile.save(os.path.join(outpath, '{:04}.png'.format(tilenum)) )


And running again...

Code Select Expand


Record at offset 768 has 3 bytes unaccounted for.
Record at offset 9359 has 3 bytes unaccounted for.
Record at offset 14366 has 39 bytes unaccounted for.
Record at offset 15087 has 39 bytes unaccounted for.
Record at offset 21646 has 131 bytes unaccounted for.
Record at offset 22560 has 39 bytes unaccounted for.
Record at offset 54601 has 963 bytes unaccounted for.
Record at offset 61481 has 259 bytes unaccounted for.
Record at offset 68227 has 259 bytes unaccounted for.
Record at offset 74973 has 39 bytes unaccounted for.
Record at offset 87197 has 259 bytes unaccounted for.
Record at offset 94720 has 259 bytes unaccounted for.
Record at offset 98099 has 259 bytes unaccounted for.
Record at offset 101737 has 39 bytes unaccounted for.
Record at offset 124062 has 259 bytes unaccounted for.
Record at offset 131326 has 259 bytes unaccounted for.
Record at offset 140144 has 259 bytes unaccounted for.
Record at offset 148185 has 259 bytes unaccounted for.
Record at offset 155708 has 259 bytes unaccounted for.
Record at offset 160382 has 259 bytes unaccounted for.
Record at offset 163761 has 259 bytes unaccounted for.
Record at offset 171284 has 259 bytes unaccounted for.
Record at offset 178030 has 259 bytes unaccounted for.
Record at offset 197467 has 259 bytes unaccounted for.
Record at offset 204731 has 259 bytes unaccounted for.
Record at offset 214326 has 259 bytes unaccounted for.
Record at offset 235317 has 259 bytes unaccounted for.
Record at offset 253718 has 259 bytes unaccounted for.
Record at offset 262277 has 259 bytes unaccounted for.
Record at offset 266692 has 259 bytes unaccounted for.
Record at offset 296888 has 531 bytes unaccounted for.
Record at offset 305411 has 243 bytes unaccounted for.
Record at offset 309719 has 243 bytes unaccounted for.
Record at offset 321156 has 899 bytes unaccounted for.
Record at offset 325931 has 243 bytes unaccounted for.
Record at offset 343340 has 531 bytes unaccounted for.
Record at offset 353794 has 515 bytes unaccounted for.
Record at offset 363293 has 259 bytes unaccounted for.
Record at offset 374943 has 259 bytes unaccounted for.
Record at offset 395189 has 171 bytes unaccounted for.
Record at offset 407614 has 171 bytes unaccounted for.
Record at offset 415826 has 259 bytes unaccounted for.
Record at offset 433431 has 259 bytes unaccounted for.
Record at offset 439659 has 259 bytes unaccounted for.
Record at offset 451922 has 259 bytes unaccounted for.
Record at offset 473916 has 147 bytes unaccounted for.
Record at offset 481227 has 515 bytes unaccounted for.
Record at offset 485490 has 243 bytes unaccounted for.
Record at offset 488542 has 147 bytes unaccounted for.
Record at offset 491213 has 3 bytes unaccounted for.
Record at offset 554159 has 259 bytes unaccounted for.
Record at offset 563460 has 903 bytes unaccounted for.
Record at offset 566188 has 259 bytes unaccounted for.
Record at offset 597635 has 259 bytes unaccounted for.
Record at offset 629082 has 1299 bytes unaccounted for.
Record at offset 650040 has 1299 bytes unaccounted for.
Record at offset 662219 has 323 bytes unaccounted for.
Record at offset 664428 has 787 bytes unaccounted for.
Record at offset 689851 has 1539 bytes unaccounted for.


Well, we are missing data. Now, is this actual data, or blank regions? Let's investigate one. Offset 54601 is record #7, and according to my debug CSV has a size of 6880. If we have 969 unaccounted bytes, that puts at offset 54601+6880-963 = 60518. And it looks like there's a header for a 24 * 40 pixel image here. This implies that we are off by 1 on the number of images in a region. However, the 3 byte misalignments look a bit too small. Sure enough, investigating one of those shows that it actually is a 0, 0, 0 header. So, let's expand to grab that last image, but provide the ability to ignore an empty image if needed.

Code Select Expand


self.numimages = self.header[0] + 1

for tilenum in range(self.numimages):
    (width, height, unknown) = struct.unpack('<3B',
        filedata.read(3))
    if width > 0 and height > 0:
        tile = Image.fromstring("P", (width, height),
            filedata.read(width*height))
        tile.putpalette(palette)
        self.images.append(self.maskimage(tile))


No more warnings, but some of these extra images seem a bit out of place. Hrm, maybe it is just garbage data after all. It doesn't hurt to decode it though.

Now for palette time! This will require delaying loading the picture data a bit later, so we can load the palette explicitly first. Here are the updates to the imagerecord class:

Code Select Expand


    def __init__(self, filedata, offset, size):
        self.offset = offset
        self.size = size

        self.images = []
        # Store the file handle for future use
        self.filedata = filedata

        if offset > 0:
            filedata.seek(offset)
            headerstruct = '<12B'
            self.header = struct.unpack(headerstruct,
                filedata.read(struct.calcsize(headerstruct)) )

            self.numimages = self.header[0] + 1

        else:
            self.numimages = 0
            self.header = []

    def loadimages(self, palette, skipimages=0):
        if self.offset > 0:
            self.filedata.seek(self.offset + 12)

            for tilenum in range(self.numimages):
                (width, height, unknown) = struct.unpack('<3B',
                    self.filedata.read(3))
                # Skip past this image if requested (i.e. for palettes)
                if skipimages > 0:
                    self.filedata.seek(width*height, os.SEEK_CUR)
                    skipimages = skipimages - 1
                elif width > 0 and height > 0:
                    tile = Image.fromstring("P", (width, height),
                        self.filedata.read(width*height))
                    tile.putpalette(palette)
                    self.images.append(self.maskimage(tile))

            # Check to see if we actually loaded all data from this record
            leftover = self.offset + self.size - self.filedata.tell()
            if leftover > 0:
                print "Record at offset {} has {} bytes unaccounted for.".format(self.offset, leftover)
            elif leftover < 0:
                print "Record at offset {} read {} bytes beyond its boundary.".format(self.offset, -leftover)

    def getpalette(self):
        """ Loads the first image in this record as a palette."""
        self.filedata.seek(self.offset + 12)
        (width, height, unknown) = struct.unpack('<3B',
            self.filedata.read(3))
        if width*height != 768:
            raise Exception('This image is not a palette!')
        else:
            return self.filedata.read(width*height)


And the main load routine:

Code Select Expand


# Create the image records using list comprehension
self.records = [imagerecord(graphicsfile, offset, size)
    for (offset, size) in zip(headerdata, headerdata2)]

palette1 = self.records[5].getpalette()
palette2 = self.records[53].getpalette()

for recnum, record in enumerate(self.records):
    if recnum == 53:
        record.loadimages(palette2, skipimages=1)
    elif recnum == 5:
        record.loadimages(palette1, skipimages=1)
    else:
        record.loadimages(palette1, skipimages=1)


Weird. Everything looks like the correct colours, just VERY dark. Either it's in some strange format, or it's an alternate palette. A quick comparison using GIMP and a Hex editor doesn't show any direct correlation. Bah. It's not worth it right now. I'll keep the code in, but I will limit it for group 53 data and leave the main data using the screenshot for the palette. Let's get to the mapping!

In order to get started on the mapping, I need to refactor how I load the map. Previously, we loaded the map byte-at-a-time to make visualization easy. Now we should load as a series of 16-bit halfwords instead. I should also add a quick routine to generate a map csv so I can see the data values for each tile easily. Before I do that, let's get the previous functionality refactored and try it on a few other maps for comparison.

Code Select Expand


import struct, sys, os
from PIL import Image

class xargonmap(object):
    def __init__(self, filename):
        # Grab the map from the file name (sans ext)
        # TODO: Maps may have embedded names. To consider
        (temppath, tempfname) = os.path.split(filename)
        (self.name, tempext) = os.path.splitext(tempfname)

        # Load the map data as a 98 x 98 array of 2-byte positions:
        mapfile = open(filename, 'rb')
        pattern = '<{}H'.format(64*128)

        self.tiles = struct.unpack(pattern,
            mapfile.read(struct.calcsize(pattern)) )
        mapfile.close()

    def debugcsv(self):
        # Remember that the map is height-first. We need to convert to
        # width-first
        pass

    def debugimage(self):
        # Turn the map data into a list of 3-byte tuples to visualize it.
        # Start by pre-creating an empty list of zeroes then copy it in
        visualdata = [None] * (64*128)
        for index in range(64*128):
            visualdata[index] = (self.tiles[index]%256, self.tiles[index]/256, 0)

        # Tell PIL to interpret the map data as a RAW image:
        mapimage = Image.new("RGB", (64, 128) )
        mapimage.putdata(visualdata)
        mapimage.rotate(-90).save(self.name + '.png')


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargonmap.py [Map File]
TODO
"""
    else:
        for filename in sys.argv[1:]:
            themap = xargonmap(filename)
            themap.debugcsv()
            themap.debugimage()


For today's effort, let's try to see if we can get better decoding of each image record. It may not be perfect, but we should be able to get good-enough decoding to get the map tiles ready for use. It's very hard to guess what every number might mean from a black box perspective, so some of those fields will just have to remain unknowns for now. Usually, the best approach is to start with a number that you think should be present and try to find it.

The most likely candidates I see are the 3rd and 4th last numbers. Since we know the tile images are 16 x 16, we see a lot of records with those numbers. What I'm going to do is split each record up into a series of images, using those dimensions for each series. We'll see how well that turns out. I'm also going to need to clean up my code a bit more to support this, so it's time to group things into classes. The updated script is below:

Code Select Expand


import struct, sys, os, pdb, csv
from PIL import Image

def createpath(pathname):
    """ Simple utility method for creating a path only if it does
    not already exist.
    """
    if not os.path.exists(pathname):
        os.mkdir(pathname)

class imagefile(object):
    def __init__(self, filename, palette):
        filesize = os.path.getsize(filename)
        graphicsfile = open(filename, 'rb')

        header = '<62L'
        headerdata = struct.unpack(header,
            graphicsfile.read(struct.calcsize(header)) )

        graphicsfile.seek(0x200)

        header2 = '<62H'
        headerdata2 = struct.unpack(header2,
            graphicsfile.read(struct.calcsize(header2)) )

        # Create the image records using list comprehension
        self.records = [imagerecord(graphicsfile, offset, size, palette)
            for (offset, size) in zip(headerdata, headerdata2) if offset > 0]

    def debug_csv(self, filename):
        with open(filename, 'wb') as csvfile:
            writer = csv.writer(csvfile)
            for recnum, record in enumerate(self.records):
                writer.writerow([recnum, record.offset, record.size] + list(record.header))

    def save(self, outpath):
        createpath(outpath)
        for recnum, record in enumerate(self.records):
            record.save(os.path.join(outpath, '{:02}-{}'.format(recnum, record.offset)))


class imagerecord(object):
    def __init__(self, filedata, offset, size, palette):
        self.offset = offset
        self.size = size

        filedata.seek(offset)
        headerstruct = '<16B'
        self.header = struct.unpack(headerstruct,
            filedata.read(struct.calcsize(headerstruct)) )

        self.width = self.header[12]
        self.height = self.header[13]

        self.images = []

        for tilenum in range(size/(self.width*self.height)):
            tile = Image.fromstring("P", (self.width, self.height),
                filedata.read(self.width*self.height))
            tile.putpalette(palette)
            self.images.append(tile)


    def save(self, outpath):
        createpath(outpath)
        for tilenum, tile in enumerate(self.images):
            tile.save(os.path.join(outpath, '{:04}.png'.format(tilenum)) )


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargongraphics.py [Graphics File]
TODO
"""
    else:
        # Use the screenshot to grab the proper colour palette:
        palimage = Image.open('screeny.png')
        palette = palimage.getpalette()

        for filename in sys.argv[1:]:
            xargonimages = imagefile(filename, palette)
            xargonimages.debug_csv('debug.csv')
            xargonimages.save('output')


So now we have an imagefile class for the overall file, and an imagerecord for each record in the file's table of contents. I also moved my save CSV code and load/save image code into functions for each class.

Anyone unfamiliar with zip, enumerate or Python classes in general can refer to the corresponding Python documentation sections linked for each. I used one yesterday, but if you are unfamiliar with list comprehensions, I recommend looking that up too. I'm sure I'll use more going forward.

The first few sets of images start to look more like something, but they're still a bit weird. Set #5, however, is where things start to get interesting. Here we have an almost-correct decoding of the player sprite!



Guess the guess wasn't too far off. Since we're shifting between each sub-image, I suspect there may be a small header between each image. Let's see what we can find. I'm going to use image set #9, because it's 16 by 16 and very easy to see image alignment. The first image seems to be shifted by 1 pixel, and the the second image seems to be shifted by 2 pixels. Hrm. 3 pixel header? Let's find this in a hex editor. My debug code nicely names the directories by their offsets, so start from offset 74973 + 16 for header + 256 for first image -1 for first image error gets me to 75244. And this decodes to:
16 16 0

So, it looks like the end of the previous header was actually the start of the first image's header. And it also appears I was accidentally grabbing the first pixel from the image. Let's go fix all this, and decode the headers for each image. Also, looking at record 9, I ended up with 47 images. The header for that record has 47 as the first number. Since I know now that we can have variable size images, I'm going to assume this is the number of images in a record.

On my first attempt, I got the following exception

Code Select Expand


Traceback (most recent call last):
  File "xargongraphics.py", line 100, in <module>
    xargonimages = imagefile(filename, palette)
  File "xargongraphics.py", line 46, in __init__
    for (offset, size) in zip(headerdata, headerdata2) if offset > 0]
  File "xargongraphics.py", line 78, in __init__
    filedata.read(width*height))
  File "/usr/lib/python2.7/dist-packages/PIL/Image.py", line 1797, in fromstring
    im.fromstring(data, decoder_name, args)
  File "/usr/lib/python2.7/dist-packages/PIL/Image.py", line 590, in fromstring
    d.setimage(self.im)
ValueError: tile cannot extend outside image


Which implies something is mixed up. Debugger time.

Code Select Expand


-> tile = Image.fromstring("P", (width, height),
(Pdb) width
1
(Pdb) height
0
(Pdb) self.header
(128, 1, 0, 0, 10, 0, 18, 0, 34, 2)
(Pdb)


Oops. I think I subtracted too many bytes when I shrunk the header. Here's the code with that fixed.

Code Select Expand


import struct, sys, os, pdb, csv
from PIL import Image

def createpath(pathname):
    """ Simple utility method for creating a path only if it does
    not already exist.
    """
    if not os.path.exists(pathname):
        os.mkdir(pathname)

class imagefile(object):
    def __init__(self, filename, palette):
        filesize = os.path.getsize(filename)
        graphicsfile = open(filename, 'rb')

        header = '<62L'
        headerdata = struct.unpack(header,
            graphicsfile.read(struct.calcsize(header)) )

        graphicsfile.seek(0x200)

        header2 = '<62H'
        headerdata2 = struct.unpack(header2,
            graphicsfile.read(struct.calcsize(header2)) )

        # Create the image records using list comprehension
        self.records = [imagerecord(graphicsfile, offset, size, palette)
            for (offset, size) in zip(headerdata, headerdata2) if offset > 0]

    def debug_csv(self, filename):
        with open(filename, 'wb') as csvfile:
            writer = csv.writer(csvfile)
            for recnum, record in enumerate(self.records):
                writer.writerow([recnum, record.offset, record.size] + list(record.header))

    def save(self, outpath):
        createpath(outpath)
        for recnum, record in enumerate(self.records):
            record.save(os.path.join(outpath, '{:02}-{}'.format(recnum, record.offset)))


class imagerecord(object):
    def __init__(self, filedata, offset, size, palette):
        self.offset = offset
        self.size = size

        filedata.seek(offset)
        headerstruct = '<12B'
        self.header = struct.unpack(headerstruct,
            filedata.read(struct.calcsize(headerstruct)) )

        self.numimages = self.header[0]
        self.images = []

        for tilenum in range(self.numimages):
            (width, height, unknown) = struct.unpack('<3B',
                filedata.read(3))

            tile = Image.fromstring("P", (width, height),
                filedata.read(width*height))
            tile.putpalette(palette)
            self.images.append(tile)


    def save(self, outpath):
        createpath(outpath)
        for tilenum, tile in enumerate(self.images):
            tile.save(os.path.join(outpath, '{:04}.png'.format(tilenum)) )


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargongraphics.py [Graphics File]
TODO
"""
    else:
        # Use the screenshot to grab the proper colour palette:
        palimage = Image.open('screeny.png')
        palette = palimage.getpalette()

        for filename in sys.argv[1:]:
            xargonimages = imagefile(filename, palette)
            xargonimages.debug_csv('debug.csv')
            xargonimages.save('output')


And the output for a random record with variable-sized images:


Nailed it!

Now, to top things off, let's mask transparent areas of sprites. It looks like colour 0 (black) is used for transparency. Let's just add a routine to loop through the image data, create a mask, then convert each to RGBA. This will make our lives much easier going forward. I'm actually going to just copy a routine I already wrote for my Shadow Caster maps.

And done...

Code Select Expand


import struct, sys, os, pdb, csv
from PIL import Image

def createpath(pathname):
    """ Simple utility method for creating a path only if it does
    not already exist.
    """
    if not os.path.exists(pathname):
        os.mkdir(pathname)

class imagefile(object):
    def __init__(self, filename, palette):
        filesize = os.path.getsize(filename)
        graphicsfile = open(filename, 'rb')

        header = '<62L'
        headerdata = struct.unpack(header,
            graphicsfile.read(struct.calcsize(header)) )

        graphicsfile.seek(0x200)

        header2 = '<62H'
        headerdata2 = struct.unpack(header2,
            graphicsfile.read(struct.calcsize(header2)) )

        # Create the image records using list comprehension
        self.records = [imagerecord(graphicsfile, offset, size, palette)
            for (offset, size) in zip(headerdata, headerdata2) if offset > 0]

    def debug_csv(self, filename):
        with open(filename, 'wb') as csvfile:
            writer = csv.writer(csvfile)
            for recnum, record in enumerate(self.records):
                writer.writerow([recnum, record.offset, record.size] + list(record.header))

    def save(self, outpath):
        createpath(outpath)
        for recnum, record in enumerate(self.records):
            record.save(os.path.join(outpath, '{:02}-{}'.format(recnum, record.offset)))


class imagerecord(object):

    @staticmethod
    def maskimage(inimage):
        """ Masks colour 0 in the given image, turning those pixels
        transparent. Returns the resulting RGBA image.
        """
        tempmask = Image.new('L', inimage.size, 255)
        maskdata = list(tempmask.getdata())
        outimage = inimage.convert("RGBA")

        for pos, value in enumerate(inimage.getdata()):
            if value == 0:
                maskdata[pos] = 0

        tempmask.putdata(maskdata)
        outimage.putalpha(tempmask)
        return outimage

    def __init__(self, filedata, offset, size, palette):
        self.offset = offset
        self.size = size

        filedata.seek(offset)
        headerstruct = '<12B'
        self.header = struct.unpack(headerstruct,
            filedata.read(struct.calcsize(headerstruct)) )

        self.numimages = self.header[0]
        self.images = []

        for tilenum in range(self.numimages):
            (width, height, unknown) = struct.unpack('<3B',
                filedata.read(3))

            tile = Image.fromstring("P", (width, height),
                filedata.read(width*height))
            tile.putpalette(palette)
            self.images.append(self.maskimage(tile))


    def save(self, outpath):
        createpath(outpath)
        for tilenum, tile in enumerate(self.images):
            tile.save(os.path.join(outpath, '{:04}.png'.format(tilenum)) )


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print """Usage: python xargongraphics.py [Graphics File]
TODO
"""
    else:
        # Use the screenshot to grab the proper colour palette:
        palimage = Image.open('screeny.png')
        palette = palimage.getpalette()

        for filename in sys.argv[1:]:
            xargonimages = imagefile(filename, palette)
            xargonimages.debug_csv('debug.csv')
            xargonimages.save('output')


A quick audit of the map tile groups show that there