Language…
19 users online:  Ahrion,  Atari2.0, autisticsceptile1993, codfish1002, eltiolavara9, Green, Heitor Porfirio, koffe190, LazyRuns, LucasRCD, Mischievous Marc, neidoodle, NewPointless, Pizzagamer9791, RicardoDeMelo,  Ringo, Serena, sinseiga, YuriGamer - Guests: 271 - Bots: 278
Users: 64,795 (2,376 active)
Latest user: mathew

SMW/ROMhacking tool related code library

  • Pages:
  • 1
  • 2
This is a thread used for the archiving of programming code that can be used in the development of SMW tools.

Unfortunately, the only language I know is C#.net, so I can only contribute C#.net code.

I'll update this post with any new code that anyone posts here.

Also, there's nothing I can do about the table stretch... D:

If such a thing exists for the programming forum, I'd like to request for sticky.

------------C#.NET---------------

1. Getting ROM data in to a byte array

This code actually allows you to get the ROM data so that you can change it.

Variables:

ROMname: A string that contains the filepath of the ROM.

Code
byte[] _rombuffer = File.ReadAllBytes(ROMname);

Thanks to Ersanio for the simpler version.

2. Writing data to a ROM file

This code will take a byte array and write it to a ROM file.

Variables:

ROMname: A string that contains the filepath of the ROM.
ROMbyte: A byte array containing the contents of the ROM.

Code
File.WriteAllBytes(ROMname, ROMbyte);


3. Patching an IPS patch

This code patches an IPS patch.

Variables:

romname: A string containing the filepath of the ROM.
patchname: A string containing the filepath of the patch.

Code
            FileStream romstream = new FileStream(romname, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite);
            FileStream ipsstream = new FileStream(patchname, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite);
            int lint = (int)ipsstream.Length;
            byte[] ipsbyte = new byte[ipsstream.Length];
            byte[] rombyte = new byte[romstream.Length];
            IAsyncResult romresult;
            IAsyncResult ipsresult = ipsstream.BeginRead(ipsbyte, 0, lint, null, null);
            ipsstream.EndRead(ipsresult);
            int ipson = 5;
            int totalrepeats = 0;
            int offset = 0;
            bool keepgoing = true;
            //////////////////End Init code
            //////////////////Start main code
            while (keepgoing == true)
            {
                offset = ipsbyte[ipson] * 0x10000 + ipsbyte[ipson + 1] * 0x100 + ipsbyte[ipson + 2];
                ipson++;
                ipson++;
                ipson++;
                /////////////split between repeating byte mode and standard mode
                if (ipsbyte[ipson] * 256 + ipsbyte[ipson + 1] == 0)
                {
                    ////////////repeating byte mode
                    ipson++;
                    ipson++;
                    totalrepeats = ipsbyte[ipson] * 256 + ipsbyte[ipson + 1];
                    ipson++;
                    ipson++;
                    byte[] repeatbyte = new byte[totalrepeats];
                    for (int ontime = 0; ontime < totalrepeats; ontime++)
                        repeatbyte[ontime] = ipsbyte[ipson];
                    romstream.Seek(offset, SeekOrigin.Begin);
                    romresult = romstream.BeginWrite(repeatbyte, 0, totalrepeats, null, null);
                    romstream.EndWrite(romresult);
                    ipson++;
                }
                else
                {
                    ////////////standard mode
                    totalrepeats = ipsbyte[ipson] * 256 + ipsbyte[ipson + 1];
                    ipson++;
                    ipson++;
                    romstream.Seek(offset, SeekOrigin.Begin);
                    romresult = romstream.BeginWrite(ipsbyte, ipson, totalrepeats, null, null);
                    romstream.EndWrite(romresult);
                    ipson = ipson + totalrepeats;
                }
                /////////////Test For "EOF"
                if (ipsbyte[ipson] == 69 && ipsbyte[ipson + 1] == 79 && ipsbyte[ipson + 2] == 70)
                    keepgoing = false;
            }
            romstream.Close();
            ipsstream.Close();



-----------Ruby 1.9-------------
Linky
Not to sound like a smartass or something, but there is an easier way to load a ROM into a byte array. It's just 1 line of code. Assume that path is the filepath to the ROM.

Code
byte[] _rombuffer = File.ReadAllBytes(path);

My blog. I could post stuff now and then

My Assembly for the SNES tutorial (it's actually finished now!)
Originally posted by Morton
Not to sound like a smartass or something, but there is an easier way to load a ROM into a byte array. It's just 1 line of code. Assume that path is the filepath to the ROM.

Code
byte[] _rombuffer = File.ReadAllBytes(path);

Oh. I had no idea about that, but thanks! Updated that method and also added File.WriteAllBytes to the list.

I'm also working on the code to create an IPS patch, if anyone cares.
Now with extra girl and extra hacker
HyperHacker
Reading/writing the entire file at once is not a good method for most programs. It's slow (lots of unnecessary disk I/O), uses more RAM than it needs to, and if you have the file also open in another program (e.g. hex editor) and change something completely unrelated to what you're working on, the change gets wiped out.
My projects - ROM hacks, Gameshark codes, homebrew, useful apps, online 65816 disassembler
SCAM: HomebreWare and Wiiunlocker
Generation over 9000: The first time you see this, put a meme in your signature.
I have edited this post from the original version. Revision log follows.
  • (2 July 2009) Added Makefile section with code to run xkas to transform .s files into .smc files.
  • (5 June 2009) Show output from newer snes-rom.rb that includes version; link to wiki page about SNES header.
  • (3 June 2009) Add find-runs.rb to find free space, or runs of repeated bytes.
  • (31 May 2009) Added a public domain dedication; added Subversion notice; added ruby-ips-2.rb and snes-rom.rb to the post; removed the chat about async I/O from the top of the post; replaced the bottom of the post with a short note below Ruby example 2.
  • (4 May 2009) Posted the original version.

Dedication to the public domain: I intend to allow the general public to use my contributions to this SMW-tool-related code library. I place the source code, that I write and embed into this post, into the public domain. Therefore, you do not need a copyright license from me to use this source code.

Subversion: My SVN repository http://opensvn.csie.org/kernigh/trunk/examples/ruby-snes/ holds some files that I link from this post. I may replace these files with newer revisions.


------------Ruby 1.9---------------

Ruby language is not popular like C#, but I use Ruby with SMW. Kaizo and Ersanio have three pieces in C#, I can redo them in Ruby 1.9.

1. Getting ROM data in to a byte array

This is the code:

Code
rombuffer = open(path, "rb") { |f| f.read }


This is an example to use my code at the interactive Ruby prompt:

Code
irb(main):001:0> path = "smw-clean.smc"
=> "smw-clean.smc"
irb(main):002:0> rombuffer = open(path, "rb") { |f| f.read }; 0
=> 0
irb(main):003:0> rombuffer.length
=> 524800
irb(main):004:0> rombuffer.index "SUPER"
=> 33216
irb(main):005:0> rombuffer[33216, 21]
=> "SUPER MARIOWORLD     "



2. Writing data to a ROM file

This is the code:

Code
open(path, "wb") { |f| f.write(rombuffer) }


This continues my example:

Code
irb(main):006:0> rombuffer[33216, 21] = "Super Ruby 1.9 World "
=> "Super Ruby 1.9 World "
irb(main):007:0> path = "smw-ruby.smc"
=> "smw-ruby.smc"
irb(main):008:0> open(path, "wb") { |f| f.write(rombuffer) }
=> 524800


Now if emulator plays smw-ruby.smc, then the title of ROM is "Super Ruby 1.9 World " (and the checksum is bad).

(31 May) This example was slow and wasteful because I used code that reads and writes the entire file, but I only needed to seek and write 21 bytes. HyperHacker (in the post before this post) explained, "Reading/writing the entire file at once is not a good method for most programs."


3. Patching an IPS patch (ruby-ips-2.rb)

Old version: ruby-ips.rb
(31 May 2009) New version: ruby-ips-2.rb (svn)

Around March 2009, I wrote ruby-ips.rb and began to use it as my normal IPS patcher before I play SMW hack. This program has two known bugs: the error messages at line 115 does not work, and Emacs does not like "-*- encoding: US-ASCII -*-" at line 2.

Around C3 of May 2009, I wrote ruby-ips-2.rb and began to use it as my normal IPS patcher. This second version fixes the known bugs and also adds -o option to copy clean ROM before apply patch. Now I do not need to copy my clean SMW ROM before I start patcher.

From command line, to apply patch.ips to rom.sfc, modifying rom.sfc:

Code
$ ./ruby-ips-2.rb -f rom.sfc patch.ips


To copy smw-clean.smc to smtko-demo1.smc then apply smtko_demo_1.ips:

Code
$ ./ruby-ips-2.rb -o smtko-demo1.smc -f smw-clean.smc smtko_demo_1.ips


This is the main loop to process the patch, where patch is the IO for the patch, file is the IO for the ROM, pread.call(3) is a lambda to read 3 bytes from patch but raise an error if there are less than 3 bytes, unpack2 and unpack3 are methods to convert bytes to numbers, hex is method to convert number to ASCII.

Code
      # process each record
      while true
        offset = pread.call(3)

	# "EOF" marks the end of an IPS patch, but check if there are
	# bytes in this patch after the "EOF".
	#
	if offset == "EOF"
	  truncation = patch.read(4)

	  if truncation == nil
	    puts "\tend of patch" if list_records

	  elsif truncation.length == 3
	    # truncate the file, as does Lunar IPS
	    truncation = unpack3(truncation)

	    if list_records
	      puts "\tend of patch, truncation at #{hex(truncation)}"
	    end
	    file.truncate(unpack3(truncation)) if filename

	  else
	    raise "unexpected data after \"EOF\" in patch\n" +
	    	  "problem with offset \"EOF\" #{hex(0x454f46)}?"
	  end

	  # break from 'while true' loop
	  break
	end

	offset = unpack3(offset)
	length = unpack2(pread.call(2))
	case length
	when 0
	  # this patch record uses run-length encoding (RLE)
	  length = unpack2(pread.call(2))
	  byte = pread.call(1)

	  if list_records
	    puts "\toffset #{hex(offset)}, length #{hex(length)}, " +
	       	 "RLE #{hex(byte.ord)}"
	  end
	  if filename
	    file.pos = offset
	    file.write(byte * length)
	  end
	else
	  # this is a normal patch record
	  if list_records
	    puts "\toffset #{hex(offset)}, length #{hex(length)}"
	  end
	  if filename
	    file.pos = offset
	    file.write pread.call(length)
	  else
	    patch.pos += length
	  end
	end # case length
      end   # while true



4. ROM expansion with clean ROM check

I have some code to expand a ROM and compute the checksum, but I cut and derived the code from a longer Ruby program that I wrote for my SMW hack, so some variable names might be strange.

This code makes an expanded copy of a ROM and also checks that the original ROM is clean Super Mario World. The clean ROM must be headerless or have a valid SMC header. CAVEAT: the expanded ROM will be headerless. (I am not sure how to make the header, because I noticed that SMW hacks have invalid SMC headers. I need to learn more about SMC headers.) Also, the expanded ROM will have a bad checksum.

This code might not be ready for the code library, because I would need to give SMC header to expanded ROM, and to separate the ROM expansion and the clean ROM check into two pieces of code.

Change clean_rom to path to clean ROM. Change expanded_rom to path to expanded ROM. Change Expanded_banks to 32 for 1 MB, 64 for 2 MB, or 128 for 4 MB. Other numbers from 17 through 127 will play in snes9x, but the ROM size in the Super NES header will wrong, because it will be the next power of 2.

Code
require 'digest/sha1'

clean_rom = "smw-ruby.smc"
expanded_rom = "expanded.sfc"

# each lorom bank has 0x8000 bytes
SMW_banks = 16
Expanded_banks = 128

# SHA1 of the clean SMW ROM (all 16 banks, without SMC header)
SMW_clean = "6b47bb75d16514b6a476aa0c73a683a2a4c18765"

smw = nil
rom = nil
begin
	# open SMW ROM image for reading; seek past any SMC header
	smw = open(clean_rom, "rb")
	size = smw.stat.size
	goal = SMW_banks * 0x8000

	if size == goal
		# headerless ROM
	elsif size >= (goal + 512)
		# check SMC header
		size = smw.read(2).unpack("v")[0] * 0x2000
		if size == goal
			# seek past header
			smw.pos = 512
		else
			raise "#{clean_rom}: wrong size in SMC header"
		end
	else
		raise "#{clean_rom}: wrong size"
	end

	rom = open(expanded_rom, "wb")

	# copy banks
	check = Digest::SHA1.new
	SMW_banks.times do
		bank = smw.read(0x8000)
		check.update(bank)
		rom.write(bank)
	end
	smw.close

	unless check.hexdigest == SMW_clean
		raise "${clean_rom}: not a clean SMW ROM"
	end

	# expand ROM
	bank = "\xff" * 0x8000
	(Expanded_banks - SMW_banks).times do
		rom.write(bank)
	end

	# Write ROM size to Super NES header. If the number of
	# banks is not a power of 2, then round upward.
	#
	byte = (Math.log2(Expanded_banks) + 5).ceil.to_i
	rom.pos = 0x7fd7
	rom.write([byte].pack "C")
	rom.close
rescue Exception => e
	smw.close if smw and not smw.closed?
	rom.close if rom and not rom.closed?

	File.delete expanded_rom	# delete bad file
	raise e
end



5. Recompute checksum in Super NES header

This is a good programming exercise. This code recomputes the checksum, but my version only works with LoROM (like Super Mario World) and only if the ROM contains a whole number of banks. If the number of banks is not a power of 2, then my code will mirror the banks, so that the checksum equals what snes9x would compute.

(Lunar Magic seems to adjust some bytes so that the ROM has the same checksum as before, instead of recomputing the checksum.)

Change target_rom to path to ROM.

Code
target_rom = "expanded.sfc"

io = nil
begin
	io = open(target_rom, "rb+")
	size = io.stat.size
	bank_count = size / 0x8000

	case size % 0x8000
	when 0
		base = 0		# no SMC header
	when 512
		base = 512		# SMC header
	else
		raise "#{rom}: wrong size"
	end

	# Compute checksum in Super NES header. Use the formula
	# from wlalink/compute.c of WLA DX, which is the same
	# as that from memmap.c of Snes9x ("from NSRT").
	#
	io.pos = base + 0x7fdc
	io.write([0xffff, 0x0000].pack "vv")

	# compute checksum of each bank
	banksum = []
	io.pos = base
	bank_count.times do |bank|
		banksum[bank] = io.read(0x8000).sum(16)
	end

	# compute checksum of all banks
	checksum = 0
	(2 ** Math.log2(bank_count).ceil).times do |bank|
		# handle the mirror banks
		bank >>= 1 while bank >= banksum.length

		checksum += banksum[bank]
	end
	checksum &= 0xffff

	# write checksum
	io.pos = base + 0x7fdc
	io.write([checksum ^ 0xffff, checksum].pack "vv")
ensure
	io.close
end



6. Examine the Super NES header (snes-rom.rb)

(31 May 2009) Full source code: snes-rom.rb (svn)

My snes-rom.rb is a program to print the information from the SNES header.

Code
$ ./snes-rom.rb                                     
Usage: snes-rom [options] rom...
    -l                               List all candidate headers


The Super NES header consists of 64 bytes inside the ROM at SNES address 0x00ffc0. The SNES header contains information like the checksum and the interrupt vectors. (5 June 2009) For more information, see a wiki page about the SNES header.

My snes-rom.rb tries to pick the best SNES header from four possible locations, and print the information from this header. (My scoring system to pick a header seems to work for my ROM images, but is more primitive than the scoring system in bsnes, mess or snes9x.)

Here is the info from when I run snes-rom.rb with my Super Mario World ROM.

Code
$ ./snes-rom.rb smw-clean.smc
smw-clean.smc:
  SMC header: yes
  SNES header: offset 0x81c0, score: 9/9
  name: "SUPER MARIOWORLD     " (21 bytes)
  status: 0x20 (slow LoROM)
  cartridge type: 0x02 (ROM and save-RAM)
  ROM size: 0x09 (512 kilobytes)
  RAM size: 0x01 (2 kilobytes)
  country: 0x01 (NTSC)
  licensee: 0x01 (Nintendo)
  version: 0x00
  complement/checksum: 0x5f25/0xa0da
  interrupt handlers (native mode):
    COP     BRK     ABORT   NMI     UNUSED  IRQ
    0x82c3  0xffff  0x82c3  0x816a  ------  0x8374
  interrupt handlers (emulation mode):
    COP     UNUSED  ABORT   NMI     RESET   IRQBRK
    0x82c3  ------  0x82c3  0x82c3  0x8000  0x82c3


Here is the part of the code that unpacks the header.

Code
class SnesHeader
  attr_reader :name, :status, :cartridge_type, :rom_size, :ram_size
  attr_reader :country, :licensee, :version, :ckcom, :cksum
  attr_reader :nvector, :evector

  class Vector
    attr_reader :cop, :brk, :abort, :nmi, :reset, :irq
    alias_method :irqbrk, :irq

    def initialize(array)
      @cop, @brk, @abort, @nmi, @reset, @irq = array
    end
  end

  def initialize(string)
    a = string.unpack "a21CCCCCCCvva4vvvvvva4vvvvvv"

    @name = a[0]
    @status = a[1]
    @cartridge_type = a[2]
    @rom_size = a[3]
    @ram_size = a[4]
    @country = a[5]
    @licensee = a[6]
    @version = a[7]
    @ckcom = a[8]
    @cksum = a[9]
    @nvector = Vector.new a[11..16]
    @evector = Vector.new a[18..23]
  end
end


Here is an example after I pasted the above code into the interactive Ruby prompt:

Code
irb(main):032:0> s64 = open("smw-clean.smc") { |f| f.pos = 0x81c0; f.read(64) }
=> "SUPER MARIOWORLD      \x02\t\x01\x01\x01\x00%_\xDA\xA0\xFF\xFF\xFF\xFF\xC3\x
82\xFF\xFF\xC3\x82j\x81\x00\x80t\x83\xFF\xFF\xFF\xFF\xC3\x82\xC3\x82\xC3\x82\xC3
\x82\x00\x80\xC3\x82"
irb(main):033:0> h = SnesHeader.new s64
=> #<SnesHeader:0x2b9f5270 @name="SUPER MARIOWORLD     ", @status=32, @cartridge
_type=2, @rom_size=9, @ram_size=1, @country=1, @licensee=1, @version=0, @ckcom=2
4357, @cksum=41178, @nvector=#<SnesHeader::Vector:0x2b9f51b0 @cop=33475, @brk=65
535, @abort=33475, @nmi=33130, @reset=32768, @irq=33652>, @evector=#<SnesHeader:
:Vector:0x2b9f5180 @cop=33475, @brk=33475, @abort=33475, @nmi=33475, @reset=3276
8, @irq=33475>>
irb(main):034:0> printf "complement/checksum: 0x%x/0x%x\n", h.ckcom, h.cksum
complement/checksum: 0x5f25/0xa0da
=> nil
irb(main):035:0> printf "reset handler: 0x%06x\n", h.evector.reset
reset handler: 0x008000
=> nil


In the above example, I looked at the SNES header of Super Mario World. I found that h.ckcom is 0x5f25 and h.cksum is 0xa0da, which matches what I see in the emulator. The reset handler is at SNES address 0x008000.


7. Find free space, or runs of repeated bytes (find-runs.rb)

(3 June 2009) Full source code: find-runs.rb (svn)

To find free space in Super Mario World, one must look for runs of repeated bytes 0xff. There might be other tools to look for free space, but I wrote a new tool.

Code
$ ./find-runs.rb
Usage: find-runs [options] file...
    -b HEX_BYTE                      Look for HEX_BYTE (default ff)
    -s                               Sort by length of run
    -t THRESHOLD                     Require THRESHOLD bytes per run


For a clean headered SMW ROM, here are areas of free space of at least 500 bytes, sorted by length:

Code
$ find-runs -st 500 smw-clean.smc
smw-clean.smc:
  offset 0x7f190: run of 4208 bytes
  offset 0x772f0: run of 3856 bytes
  offset 0x37738: run of 2760 bytes
  offset 0x3e96e: run of 2194 bytes
  offset 0x34b63: run of 1693 bytes
  offset 0x1bc02: run of 1534 bytes
  offset 0x3a378: run of 1160 bytes
  offset 0x2de46: run of 954 bytes
  offset 0x1e25c: run of 932 bytes
  offset 0x3fe90: run of 880 bytes
  offset 0x2713e: run of 834 bytes
  offset 0x5ff0c: run of 756 bytes
  offset 0x6f28a: run of 630 bytes
  offset 0x223b6: run of 586 bytes
  offset 0x1ffe0: run of 544 bytes


My code might be too slow, because it requires 2 to 3 seconds of my computer time to look through a file of only 512 kilobytes (like Super Mario World). The first version required 8 to 10 seconds, before I changed file.read(1) to file.getc. This change caused Ruby to fill a buffer, instead of doing one system call per byte.

Here is the code to find the runs, where file is opened to read in binary mode, free_byte is the byte to look for, threshold is the minimum length to report.

Code
    while char = file.getc
      offset += 1

      if char.ord == free_byte
        # save the offset of this first byte
        first_offset = offset
        run_length = 1

        # look for more bytes
        catch(:end_of_run) do
          while char = file.getc
            offset += 1

            if char.ord == free_byte
              run_length += 1
            else
              throw :end_of_run
            end
          end
        end

        if run_length >= threshold
          # ... code to report this run ...
        end
      end
    end



------------Makefile---------------

This section is for the make(1) tool from BSD, GNU/Linux and Unix systems.

1001. Run xkas to transform .s files into .smc files

I have some xkas patches with the .s suffix. I want to test each patch with a clean ROM of Super Mario World. I want a different ROM for each patch. I also want to use the save-RAM with "all exits (Star 96)" from SMWC's Official SRAM/Save State Archive, so that I can test any level.

This Makefile copies the clean ROM, runs xkas to apply to patch, and copies the save-RAM. Here is the entire Makefile.

Code
ROM=../roms/smw-clean.smc
SRM=../distfiles/clean_smw.SRM
XKAS=xkas

all:
	@for i in *.s; do				\
	  if test -e "$${i}"; then			\
	    j="$${i%.s}.smc";				\
	    make "$${j}";				\
	  fi;						\
	done

.SUFFIXES: .smc .s

.s.smc:
	@rm -f "$@"
	@cat < "${ROM}" > "$@"
	xkas "$@" "$<"
	@i="$@"; j="$${i%.smc}.srm"; cat < "${SRM}" > "$${j}"

clean:
	@for i in *.smc; do				\
	  if test -e "$${i}"; then			\
	    j="$${i%.smc}.srm";				\
	    echo rm -f \""$${i}"\" \""$${j}"\";		\
            rm -f "$${i}" "$${j}";			\
	  fi;						\
        done



Attention! I run xkas v0.12, so I need to use ''xkas "$@" "$<"''. Some other SMW hackers run xkas v0.06, which reverses the order of the arguments, so they would need to use ''xkas "$<" "$@"''.


If I run make force-balloon-up.smc, then this Makefile copies the clean ROM to force-balloon-up.smc, applies the patch force-balloon-up.s to the new ROM, and also copies the "all exits (Star 96)" save-RAM to force-balloon-up.srm.

If I edit force-balloon-up.s and run again make force-balloon-up.smc, then this Makefile discards the ROM and the save-RAM, starts again from the clean ROM, and creates again the ROM and the save-RAM.

If I run make clean, then this Makefile deletes every .smc file and every matching .srm file in this directory. (I must put the clean ROM in a different directory.) If I run make, then this Makefile creates the ROM and the save-RAM for every .s patch in this directory.

Hacking Super Mario World since 28 February 2009
SMWDISC
I thank you for the giant hunk of code, Kernigh.

Linked to it in the first post.

Originally posted by HyperHacker
Reading/writing the entire file at once is not a good method for most programs. It's slow (lots of unnecessary disk I/O), uses more RAM than it needs to, and if you have the file also open in another program (e.g. hex editor) and change something completely unrelated to what you're working on, the change gets wiped out.


If that's a problem, then you can use this code:

Variables:

My_Rom_Path: a string containing the filepath of the ROM.
My_Offset: an integer containing the offset at the start of the read/write.
My_Length: an integer containing the number of bytes to read/write from My_Offset.
Code
            FileStream MyStream = new FileStream(My_Rom_Path, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite);
            byte[] Rom_array = new byte[My_Length];
            MyStream.Read(Rom_array, My_Offset, My_Length);
            //Insert code that edits the ROM here...
            MyStream.Write(Rom_array, My_Offset, My_Length);
            MyStream.Close();
Originally posted by Kaizo
I thank you for the giant hunk of code, Kernigh.

Linked to it in the first post.


Because you linked the first post to my post, I will edit my post when I add more code.

My next code is not ready. I wrote some code that would read *.bin/*.pal files like ExGFX. To check my code, I saved the graphics in PPM format, a simple image format. Then I can open the PPM image in Krita. If I want to post the image here, then Krita can scale by 300% and save as PNG for upload to TinyPic.

Using my current code, here is the conversion for ExGFXFFB.bin and smw4.pal from SMB3 Overworld Stuff:



The colors are wrong! This is actually my second image, after the first image was too dark. I like this image because I can already see number signs, a toad house, a fortress, a destroyed castle, a pipe. My problem is my wrong use of the palette data. I might soon have a fix.

Hacking Super Mario World since 28 February 2009
SMWDISC
grr... I've been working on code to create an IPS patch for a long time now, and I STILL CAN'T GET IT!

What happens is that it takes about a minute and a half to make the patch, and then when I try to patch it, LIPS gives me an error.

Here's the current code:

Variables:

Rom: A byte array with the un-clean ROM in it.
Crom: A byte array with the clean ROM, the same length as the un-clean ROM.

Code
            byte[] finpatch = 
            {
                0x50, 0x41, 0x54, 0x43, 0x48
            };
            int repeats = 0;
            int offset = 0;
            byte[] vars = new byte[5];
            for (int onbyte = 0; onbyte < crom.Length; onbyte++)
            {
                if (rom[onbyte] != crom[onbyte])
                { //If the two bytes are different, go into long thing.
                    offset = onbyte;
                    for (repeats = 0; (rom[onbyte + repeats] != crom[onbyte + repeats]); repeats++)
                    {
                        //find out how many repeats there are
                    }
                    //Can't be more than FFFF bytes long!
                    if (repeats > 0xFFFF)
                        repeats = 0xFFFF;
                    byte[] temp = new byte[repeats];
                    //For loop assemballing temp array
                    for (int myint = 0; myint < repeats; myint++)
                    {
                        temp[myint] = rom[onbyte];
                        onbyte++;
                    }
                    //Prepare vars variables
                    string repeats_s = string.Format("{0:X}", repeats);
                    while (repeats_s.Length < 4)
                        repeats_s = "0" + repeats_s;
                    string offset_s = string.Format("{0:X}", offset);
                    while (offset_s.Length < 6)
                        offset_s = "0" + offset_s;
                    //Assemble Vars
                    if (bytesame(temp) && temp.Length > 4)
                    {
                        //repeating byte mode
                        vars = new byte[7];
                        vars[0] = (byte)((stringtohex(offset_s.Substring(0, 1)) * 0x10) + stringtohex(offset_s.Substring(1, 1)));
                        vars[1] = (byte)((stringtohex(offset_s.Substring(2, 1)) * 0x10) + stringtohex(offset_s.Substring(3, 1)));
                        vars[2] = (byte)((stringtohex(offset_s.Substring(4, 1)) * 0x10) + stringtohex(offset_s.Substring(5, 1)));
                        vars[3] = 0;
                        vars[4] = 0;
                        vars[5] = (byte)((stringtohex(repeats_s.Substring(0, 1)) * 0x10) + stringtohex(repeats_s.Substring(1, 1)));
                        vars[6] = (byte)((stringtohex(repeats_s.Substring(2, 1)) * 0x10) + stringtohex(repeats_s.Substring(3, 1)));
                        byte thebyte = temp[0];
                        temp = new byte[1];
                        temp[0] = thebyte;
                    }
                    else
                    {
                        //standard mode
                        vars = new byte[5];
                        vars[0] = (byte)((stringtohex(offset_s.Substring(0, 1)) * 0x10) + stringtohex(offset_s.Substring(1, 1)));
                        vars[1] = (byte)((stringtohex(offset_s.Substring(2, 1)) * 0x10) + stringtohex(offset_s.Substring(3, 1)));
                        vars[2] = (byte)((stringtohex(offset_s.Substring(4, 1)) * 0x10) + stringtohex(offset_s.Substring(5, 1)));
                        vars[3] = (byte)((stringtohex(repeats_s.Substring(0, 1)) * 0x10) + stringtohex(repeats_s.Substring(1, 1)));
                        vars[4] = (byte)((stringtohex(repeats_s.Substring(2, 1)) * 0x10) + stringtohex(repeats_s.Substring(3, 1)));
                    }
                    //put it all together
                    finpatch = appendarray(appendarray(finpatch, vars), temp);
                }
            }
            byte[] EOF = 
            {
                0x45, 0x4f, 0x46
            };
            appendarray(finpatch, EOF);


Here's some of the functions I call in that code.

Code
        private int stringtohex(string input)
        {
            switch (input)
            {
                case "0":
                    return 0;
                case "1":
                    return 1;
                case "2":
                    return 2;
                case "3":
                    return 3;
                case "4":
                    return 4;
                case "5":
                    return 5;
                case "6":
                    return 6;
                case "7":
                    return 7;
                case "8":
                    return 8;
                case "9":
                    return 9;
                case "A":
                    return 0xA;
                case "B":
                    return 0xB;
                case "C":
                    return 0xC;
                case "D":
                    return 0xD;
                case "E":
                    return 0xE;
                case "F":
                    return 0xF;

            }
            return 0x10;
        }

        private byte[] appendarray(byte[] partone, byte[] parttwo)
        {
            MemoryStream appendstream = new MemoryStream();
            appendstream.Write(partone, 0, partone.Length);
            appendstream.Write(parttwo, 0, parttwo.Length);
            byte[] endbyte = new byte[partone.Length + parttwo.Length];
            appendstream.Seek(0, SeekOrigin.Begin);
            appendstream.Read(endbyte, 0, endbyte.Length);
            appendstream.Close();
            return endbyte;
        }

        private bool bytesame(byte[] bytearraytocheck)
        {
            for (int myint = 1; myint < bytearraytocheck.Length; myint++)
            {
                if (bytearraytocheck[myint] != bytearraytocheck[myint - 1])
                    return false;
            }
            return true;
        }
Originally posted by Kaizo
What happens is that it takes about a minute and a half to make the patch...


Your code is much too slow! You might want more speed before you fix the other problems with your code. So I tried to check your code for slowness.

Your function appendarray might be the major source of slowness, because it always copies the array into a new byte[...]. For example, smtko_demo_1.ips (from Super Mario TKO (demo 1)) has a size of about 994K. If your code would make this patch, then finpatch would contain 497K on average. To make each patch record (in repeating-byte mode or standard mode), you twice call finpatch. This TKO patch has about 2500 records, so you would 5000 times call finpatch. You would copy about 497K times 5000 equals about 2427 MB. I conclude that your code would copy about 2.37 GB between memory locations to generate a 994K patch for a 4 MB ROM.

You would need to replace byte[] finpatch with a smart array that keeps extra capacity and can grow larger without making 5000 extra copies. I am no C# hacker, but I wandered around Microsoft's documentation, and I guess that System.Collections.Generic.List<byte> is a smart array. The method AddRange might append another array to a smart array, like so:

Code
// This might not be valid C# code...
System.Collections.Generic.List<byte> finpatch = ...;
byte[] stuffToAppend = ...;

finpatch.AddRange(
  new System.Collections.Generic.List<byte>(stuffToAppend));


Instead of System.Collections.Generic.List, I think that MemoryStream might also work, because Microsoft says that MemoryStream is resizable. You already use MemoryStream, but I mean to use MemoryStream instead of byte[] for finpatch, like so:

Code
// This might not be valid C# code...
MemoryStream finpatch = new MemoryStream();
byte[] stuffToAppend = ...;

finpatch.Write(stuffToAppend, 0, stuffToAppend.Length);


Use something like System.Collections.Generic.List or MemoryStream instead of your slow function appendarray, and I hope that your code will not be slow.


I tried to check the remainder of your code, but (possibly because I know not C#) I am not sure why it makes an incorrect IPS patch. I only found one possible problem, in here:

Code
// Kaizo's code

for (repeats = 0; (rom[onbyte + repeats] != crom[onbyte + repeats]); repeats++)
{
    //find out how many repeats there are
}
//Can't be more than FFFF bytes long!
if (repeats > 0xFFFF)
    repeats = 0xFFFF;


So this code loops until rom[onbyte + repeats] == crom[onbyte + repeats]. If the unequal bytes repeat to the end of the array, then I suppose that the array accesses will become out of bounds. You would need to add a check that onbyte + repeats is less than the length of the arrays (unless you already have this check).

Other than this, I failed to find your problem. Maybe another user (perhaps someone who knows C#) can find the problem.

If you upload an IPS patch made from your code, then I might run the IPS patch through my patcher ruby-ips-2.rb and try to investigate why the patch is bad.

Hacking Super Mario World since 28 February 2009
SMWDISC
*FacePalms*

How could I possibly forget about MemoryStream?

...


So, I took your advice and used a MemoryStream instead of appendarray, and you were right! That was the source of the slowdown! The entire code runs in less that a split second.

Originally posted by Kernigh
I tried to check the remainder of your code, but (possibly because I know not C#) I am not sure why it makes an incorrect IPS patch. I only found one possible problem, in here:

Code
// Kaizo's code
for (repeats = 0; (rom[onbyte + repeats] != crom[onbyte + repeats]); repeats++)
{
    //find out how many repeats there are
}
//Can't be more than FFFF bytes long!
if (repeats > 0xFFFF)
    repeats = 0xFFFF;



So this code loops until rom[onbyte + repeats] == crom[onbyte + repeats]. If the unequal bytes repeat to the end of the array, then I suppose that the array accesses will become out of bounds. You would need to add a check that onbyte + repeats is less than the length of the arrays (unless you already have this check).


I already considered that possibility; I have implemented a check that breaks the loop if onbyte + repeats is greater than or equal to the length of the arrays.

Also, before, the patch wouldn't patch by LIPS, and now, LIPS patches it, it runs in an emulator, AND it opens in LM. The only problem is that when patched, there is a bad checksum, as where the one made by LIPS doesn't.


So, thanks!


Now with extra girl and extra hacker
Originally posted by Kaizo
Also, before, the patch wouldn't patch by LIPS, and now, LIPS patches it, it runs in an emulator, AND it opens in LM. The only problem is that when patched, there is a bad checksum, as where the one made by LIPS doesn't.


The two patched ROM images would must be equal at every byte. You might want to find which bytes are different, because I think that the different checksum implies that part of the ROM is corrupt.



Meanwhile, I made some progress with ppm-from-snes.rb, but I am not ready to post this program, which would convert graphics from SNES 4bpp (*.bin/*.pal/*.tpl) to PPM (*.ppm) format. My program is not ready because I want to add code for MAP16. I guess that there is some way to use MAP16 to match tiles with palettes.

I have readied two other programs, and edited my earlier post to include them. The first program is ruby-ips-2.rb (svn), a second version of my program to apply IPS patches. I made this version during C3 of May 2009. (I did not present the code to C3.) I added a feature to copy clean ROM before apply patch, so that I need not copy my clean SMW ROM before I run patcher. This helps because I obtained 19 different IPS patches at C3.

The second program is snes-rom.rb (svn), to print information about the internal SNES header in each ROM, like so:

Code
smw-clean.smc:
  SMC header: yes
  SNES header: offset 0x81c0, score: 9/9
  name: "SUPER MARIOWORLD     " (21 bytes)
  status: 0x20 (slow LoROM)
  cartridge type: 0x02 (ROM and save-RAM)
  ROM size: 0x09 (512 kilobytes)
  RAM size: 0x01 (2 kilobytes)
  country: 0x01 (NTSC)
  licensee: 0x01 (Nintendo)
  complement/checksum: 0x5f25/0xa0da
  interrupt handlers (native mode):
    COP     BRK     ABORT   NMI     UNUSED  IRQ
    0x82c3  0xffff  0x82c3  0x816a  ------  0x8374
  interrupt handlers (emulation mode):
    COP     UNUSED  ABORT   NMI     RESET   IRQBRK
    0x82c3  ------  0x82c3  0x82c3  0x8000  0x82c3
smtko-demo1.smc:
  SMC header: yes
  SNES header: offset 0x81c0, score: 9/9
  name: "SUPER MARIOWORLD     " (21 bytes)
  status: 0x20 (slow LoROM)
  cartridge type: 0x02 (ROM and save-RAM)
  ROM size: 0x0c (4 megabytes)
  RAM size: 0x03 (8 kilobytes)
  country: 0x01 (NTSC)
  licensee: 0x01 (Nintendo)
  complement/checksum: 0x5f25/0xa0da
  interrupt handlers (native mode):
    COP     BRK     ABORT   NMI     UNUSED  IRQ
    0x82c3  0xffff  0x82c3  0x816a  ------  0x8374
  interrupt handlers (emulation mode):
    COP     UNUSED  ABORT   NMI     RESET   IRQBRK
    0x82c3  ------  0x82c3  0x82c3  0x8000  0x82c3
mario-paint.sfc:
  SMC header: no
  SNES header: offset 0x7fc0, score: 7/9
  name: "MARIOPAINT           " (21 bytes)
  status: 0x20 (slow LoROM)
  cartridge type: 0x02 (ROM and save-RAM)
  ROM size: 0x0a (1 megabyte)
  RAM size: 0x05 (32 kilobytes)
  country: 0x00 (NTSC)
  licensee: 0x01 (Nintendo)
  complement/checksum: 0xb461/0x4b9e
  interrupt handlers (native mode):
    COP     BRK     ABORT   NMI     UNUSED  IRQ
    0x80ae  0x0000  0x0000  0x80d4  ------  0x80af
  interrupt handlers (emulation mode):
    COP     UNUSED  ABORT   NMI     RESET   IRQBRK
    0x0000  ------  0x0000  0x0000  0x8000  0x0000
n-warp-daisakusen-1.0.sfc:
  SMC header: no
  SNES header: offset 0xffc0, score: 9/9
  name: "N-Warp Daisakusen    " (21 bytes)
  status: 0x31 (fast HiROM)
  cartridge type: 0x00 (only ROM)
  ROM size: 0x0b (2 megabytes)
  RAM size: 0x00 (zero)
  country: 0x01 (NTSC)
  licensee: 0x33 (id "\\\xAAO\xC0")
  complement/checksum: 0x79de/0x8621
  interrupt handlers (native mode):
    COP     BRK     ABORT   NMI     UNUSED  IRQ
    0xffa0  0xffa0  0xffa0  0xffb6  ------  0xffba
  interrupt handlers (emulation mode):
    COP     UNUSED  ABORT   NMI     RESET   IRQBRK
    0xffa0  ------  0xffa0  0xffa0  0xffa1  0xffa0


I can use snes-rom.rb to see how SMW hacks increase the ROM size. The SMW hack smtko-demo1.smc has more ROM and more save-RAM than the clean ROM smw-clean.smc. My snes-rom.rb can also handle ROMs like Mario Paint and N-Warp Daisakusen, but it might have trouble with ROMs that have unusual layout or incomplete SNES header.

My computer does not run GoodTools nor NSRT, so snes-rom.rb provides my first easy way to examine a ROM, without starting an emulator, and to check if a ROM is LoROM or HiROM and if a ROM has a SMC header. I find that some ROMs have no SMC header but come with *.smc suffix, so I rename such ROMs to *.sfc to not imply a header.

Hacking Super Mario World since 28 February 2009
SMWDISC
Originally posted by Kernegh
I have readied two other programs, and edited my earlier post to include them. The first program is ruby-ips-2.rb (svn), a second version of my program to apply IPS patches. I made this version during C3 of May 2009. (I did not present the code to C3.) I added a feature to copy clean ROM before apply patch, so that I need not copy my clean SMW ROM before I run patcher. This helps because I obtained 19 different IPS patches at C3.


Wait, you did that too?

I just finished this, a simple console app that does auto clean rom copy, ips patching, and runs the rom in an emulator, as long as the cfg is set up correctly. Source code is included.
Now with extra girl and extra hacker
I actually felt like stickying this for good measure.
My blog. I could post stuff now and then

My Assembly for the SNES tutorial (it's actually finished now!)
Originally posted by Ersanio
I actually felt like stickying this for good measure.

:D

...aaaand to add some content to this post, here's some code that will convert between PC and SNES RGB values. (from after my embarrassing incident with the data repository)

Code
        private int convSnesRBG(int SnesValue)
        {
            int fin = 0;
            fin += ((SnesValue & 31) * 8) << 16;
            fin += (((SnesValue >> 5) & 31) * 8) << 8;
            fin += ((SnesValue >> 10) & 31) * 8;
            return fin;
        }



Code
        private int convPcRGB(int PcValue)
        {
            int fin = 0;

            // make 5-bit color values
            // blue
            int blue = (PcValue & 0xF8) >> 3;

            // green
            int green = (PcValue & 0xF800) >> 11;

            // red
            int red = (PcValue & 0xF80000) >> 19;

            // make a bgr color by shifting
            int bgr = blue << 10;
            bgr += green << 5;
            bgr += red;

            return fin;
        }


The second one was a good deal harder than the first one to make, so I broke it down and added comments for good measure.
Here is a very small code in C# to devide something (panel, picturebox, whatever you want) in a 16x16 raster. I used this with the MouseClick event:

Code
int location = (e.X >> 4) + ((e.Y >> 4) * 0x10);

e.X being the event argument mouse X position and guess what e.Y is.
This is useful for when making a palette editor. Here is a graphical example of how the raster coordinates are laid out:

(Yes I converted the values to hex manually and that blue grid isn't related to this.)
My blog. I could post stuff now and then

My Assembly for the SNES tutorial (it's actually finished now!)
Computation of SNES checksum can be much simpler than my code from 4 May 2009. For a headerless ROM image of Mario Paint (JU) or Super Mario World (U), said computation requires one and only one line of Ruby code.

open(image, "rb") { |rom| rom.read }.sum(16)

Code
irb(main):001:0> image = "SMW/super-mario-world.sfc"
=> "SMW/super-mario-world.sfc"
irb(main):002:0> open(image, "rb") { |rom| rom.read }.sum(16).to_s 16
=> "a0da"
irb(main):003:0> image = "mario-paint.sfc"
=> "mario-paint.sfc"
irb(main):004:0> open(image, "rb") { |rom| rom.read }.sum(16).to_s 16
=> "4b9e"


Checksum of SMW is xa0da. Checksum of Mario Paint is x49be.

If the size of the ROM is not a power of 2, then you might to mirror part of the ROM, including the mirror in the checksum. I am not sure how to do the mirror, because the formulas in Snes9x and in WLA DX seem to disagree. My code from 4 May 2009 tries to mirror some banks, but this attempt might be wrong.


Also, my dump-tms.rb script, which dumps transmusical strings in Mario Paint, also seems to work with Super Mario World. The script needs a headerless ROM. I might later post the script.

Code
$ mpdisc/dump-tms.rb -a 0e8000,0f8000 \
>   roms/SMW/super-mario-world.sfc
string $0e8000:
  $0500..$133d: from $0e8004..$0e8e42, download 3646 bytes
  $5570..$5fda: from $0e8e46..$0e98b1, download 2667 bytes
  $1360..$297c: from $0e98b5..$0eaed2, download 5661 bytes
  $0500: jump   (string ends at $0eaed6)
string $0f8000:
  $8000..$804f: from $0f8004..$0f8054, download 80 bytes
  $8100..$f01f: from $0f8058..$0fef78, download 28448 bytes
  $0500: jump   (string ends at $0fef7c)


Hacking Super Mario World since 28 February 2009
SMWDISC
Heh, does this mean I accidentally made a checksum recomputer? I just needed an AND 0xFFFF, change the format from long to ushort, and my tool was all set. lulz

Code
//Code to compute checksum in C#
//Note: this does NOT deal with ROM headers. It could go ahead and sum the header bytes.
//Note2: I know the code is not that good but atleast it does the wanted function

                Array.Clear(_rombuffer, 0, _rombuffer.Length); //clear stuff from previous run
                result = 0; //same as above
                _rombuffer = File.ReadAllBytes(ofd.FileName);
                for (int x = 0; x < _rombuffer.Length; x++)
                {
                    result += (ushort)_rombuffer[x]; //sum the bytes one by one.
                }
                result &= 0xFFFF; //filter unnecessary digits


Where _rombuffer is a byte array (byte[]), which holds the bytes of the ROM, and result being an 16-bit unsigned number (ushort), holding the result of the calculation.

I'll PERHAPS post a faster code later.
My blog. I could post stuff now and then

My Assembly for the SNES tutorial (it's actually finished now!)
Originally posted by Ersanio
Code
Array.Clear(_rombuffer, 0, _rombuffer.Length);
  //clear stuff from previous run
result = 0; //same as above
_rombuffer = File.ReadAllBytes(ofd.FileName);


The Array.Clear might not be necessary, because (if C# is like Java, Perl, Ruby) I guess that File.ReadAllBytes returns a new array.

Originally posted by Ersanio
//Note: this does NOT deal with ROM headers. It could go ahead and sum the header bytes.


You can always use a headerless ROM image, because the SMC header is an annoyance. Or in this case, with an open file handle, you can seek past the first $200 bytes and read everything else. The BSNES method of detect header is to simply check if ROM size modulo $8000 equals $200.

Here is Ruby program to print checksum of headered or headerless image.

Code
#!/usr/bin/env ruby

ARGV.each do |filename|
  open(filename, "rb") do |rom|
    header = rom.stat.size & 0x7fff == 0x200
    rom.pos = 0x200 if header
    checksum = rom.read.sum 16
    printf "%s: checksum is $%04x\n", filename, checksum
  end
end


Here is same program in Perl.

Code
#!/usr/bin/env perl

use strict;
use warnings;
use Fcntl qw(SEEK_SET);
use List::Util qw(sum);
  
for my $filename (@ARGV) {
  open my $rom, '<', $filename or die "$filename: $!";
  my $header = ((stat $rom)[7] & 0x7fff) == 0x200;
  seek $rom, 0x200, SEEK_SET if $header;
  my $checksum = sum(unpack "C*", do { local $/; <$rom>}) & 0xffff;
  printf "%s: checksum is \$%04x\n", $filename, $checksum;
}


I run both programs.

Code
~/park/snes $ ./sum.rb roms/SMW/{super-mario-world.sfc,smw-clean.smc}
roms/SMW/super-mario-world.sfc: checksum is $a0da
roms/SMW/smw-clean.smc: checksum is $a0da
~/park/snes $ ./sum.pl roms/SMW/{super-mario-world.sfc,smw-clean.smc}
roms/SMW/super-mario-world.sfc: checksum is $a0da
roms/SMW/smw-clean.smc: checksum is $a0da


The Perl version is slower, I guess that unpack is slow? Ruby has String#sum but I found no equivalent in Perl.

Ruby => Perl
rom.read => do { local $/; <$rom> }
string.sum 16 => no equivalent
string.unpack("C*").reduce(:+) => reduce { $a + $b } unpack "C*", string
no equivalent => sum unpack "C*", string

This simple formula is valid if the ROM size is a power 2 (or equivalently, if the ROM size matches a possible value for SNES address $00ffd7 in the SNES header).

If the ROM size is not a power of 2, then Snes9X, WLA DX and uCON64 disagree the checksum. They use three different formulas. I checked! Our simple formula cannot be more wrong than any of difficult formulas in Snes9x, WLA DX and uCON64. We can keep our simple formula. Any correct ROM size is a power of 2, and all four formulas agree when the ROM size is a power of 2.

I checked the formulas by reading the source code, not by running the programs. I checked the formulas at
* CMemory::Checksum_Calculate in memmap.cpp of Snes9x-GTK 74,
* compute_snes_checksum in wlalink/compute.c of WLA DX 9.5a,
* snes_chksum in src/console/snes.c of uCON64 2.0.0.

Hacking Super Mario World since 28 February 2009
SMWDISC
Sorry for the bump, but what is the code for patch a ips patch in VB2008?
Background image by Tzadkiel & Anomalin
Uh, we don't have one unless someone feels like contributing it here. Just wait and you'll see (or not).
My blog. I could post stuff now and then

My Assembly for the SNES tutorial (it's actually finished now!)
  • Pages:
  • 1
  • 2