VGMaps

Main Boards => VGMaps Social Board => Topic started by: DarkWolf on October 24, 2011, 09:52:48 am

Title: Looking for some help testing a project
Post by: DarkWolf on October 24, 2011, 09:52:48 am
Remember a long time ago I said something about one day building a distributed tool for pngout?  Well, I've actually got a very early version of it done.  I was wondering if you guys would be willing to upload images so I can collect data on pngout's performance. I'll have to make an account for you since I've got it closed to the public right now.  Please PM me if you are interested.

More info about the utility:

* API based, will allow you to create your own scripts or tools.  Uses HTTP POST and JSON, no SOAP or anything complicated required.
* First does single-pass of image using best color depth and 0 filter or palette images, 5 for RGB
* After are images have gotten their single pass, servers work on running combinations of filters and block sizes
* Simple web-based interface supports multi-file upload or web-based upload from URL
Title: Re: Looking for some help testing a project
Post by: Revned on October 24, 2011, 10:14:11 am
Interesting. Are you just starting up different pngout tasks with varying parameters? How distributed is it currently?
Title: Re: Looking for some help testing a project
Post by: DarkWolf on October 24, 2011, 10:21:27 am
Yes, right now it's just a brute force loop through various settings.  I have the worker daemons sending data to the main server when they complete an image, so hopefully I can find some shortcuts.  Right now I only have one machine doing actual work (aside from the web server).  I need to make sure the daemon I wrote is stable enough before I start installing it on other machines.  The plan in the future is to allow others to download a client to help with the work, but that's a little ways off.
Title: Re: Looking for some help testing a project
Post by: Maxim on October 24, 2011, 11:04:02 am
There's often gains to be made from compressing a 4-bit image as 8-bit (or even 24 bit for small images), so that's an extra dimension to add.

It'd be good to get something that can be run locally, provided it was clever enough to avoid running two instances against the same file at the same time. Does your current scheme scale to multiple processors on the same machine? One way to do that would be to have multiple copies of the file under attack, although it'd also be good to avoid transferring it more than once.
Title: Re: Looking for some help testing a project
Post by: DarkWolf on October 24, 2011, 11:30:28 am
Hrmm, I hadn't thought about messing with the bit-depth for palette images.  I can add that into an additional round of testing.  Right now the worker daemons can fork themselves, and work on multiple files.  Although this is a different image per-process id.  Each process has an expiration time in the event a worker disappears without ever reporting the results of work. But splitting up the work for a single image isn't out of the question.  Perhaps the workers could run a combination of the two depending on the status of the work pool.

When the time comes to make a friendly windows version of the worker, the ability to go through local files could be an option.  You'd just skip the logic that requests work from the server and run it through the same system calls to pngout.

There's a lot of work to be done still on the utility.  But I was getting excited and wanted to do some testing :)

By the way, the policy right now is to keep the images on the server for a few days after they are processed.  You could use this utility as it stands now to upload submissions to the site.  Just give Jon the URL for the image when it's done.
Title: Re: Looking for some help testing a project
Post by: Revned on October 24, 2011, 01:26:31 pm
The -r option can also help slightly, but it is quite expensive for such a small return.
Title: Re: Looking for some help testing a project
Post by: DarkWolf on October 24, 2011, 01:53:26 pm
I've already made some tweaks to the logic.  I added in the bit depth tests for /c3 images that Maxim suggested, and I reorganized the tests to only try different block sizes after the best filter was found (this came from some talk on the forum at http://www.jonof.id.au/).