Pyglet and tiles

This is my second attempt to move the graphics to the hardware-acceleration friendly pyglet module.  The thing that held me up the first time is a thing that I'm currently making headway on right now: writing a tile engine for it.

pyglet comes with built-in support for sprites, and in fact to get good use out of it it's important to use its sprite class instead of pushing images to the screen yourself, which I had been doing with Pygame.  pyglet's sprite rendering loop takes care of queueing its "official" sprites and sending them to OpenGL, and the video card, for you, so as long as you use its sprite class to display images.  The word is this is worth a substantial boost when drawing.

But pyglet doesn't provide as simple a mechanism for drawing tiles, meaning I have to simulate them with a field of sprites.  I'm doing this with an array of sprites that more-or-less mirrors those onscreen.  As the map scrolls around and new tiles enter the screen, the sprites that scrolled off are moved to the other side of the screen and updated to reflect the new map section.

Or at least that's the plan.  The problem is keeping the coordinate translations between the screen, the sprite frame, and the map all straight in my head, which has always been kind of a problem for me, probably due to some dyslexia.  Still, it seems to be coming along well for now.

Ways to speed up yer Python code

So I've been looking into ways to get Python code to run quickly, which is a continuation of the search I did last year (yikes, it's been almost a year since this began...)

I spent time some getting Cython working, and getting the code to work with it, but I kept running into difficult-to-solve problems with Python data types in Numpy arrays. Particularly, an array that wouldn't compile because the compiler claimed it was of type long when I defined it as int, or at least as far as I can tell. I eventually decided to shelve that and look in other directions.

Currently, I'm changing the Pygame bits over to Pygame in order to make use of hardware acceleration, which is iffy under Pygame. This has the additional benefit of allowing the code to work under PyPy, the successor to Psyco. A drawback, however, is that pyglet wants to run the program's main event and draw loops itself. I've gotten in some more coding experience, so I'm less standoffish about that now than I was before, but to really do things the pyglet way everything has to be a sprite, or a polygon. The way we draw liquids at the moment in Pygame uses filled polygons, so it's good that there's a way to port this over. The combination of hardware acceleration and JIT compiling is a potent one, and could potentially bring better performance than even using Psyco... if only I could figure out how to install pyglet to a PyPy installation.

If this fails for some reason, there's still weave, a system for inlining C code in Python. My problems with C and C++ tend to concern linking; C itself I have no trouble with, but getting code more complex than a handful of simply-included source files to link together into an executable has been a woeful journey for me.

Anyway, work continues. You can't see it on your end, but behind the scenes of this blog's inscrutable periods of silence are me wringing my hands over the problems I've been facing. I have considered opening the source, or at least posting bits of it, and asking for advice from you guys. I don't suppose there are any Python mavens reading this?

Figured that out

The problem I mentioned in the last post?  Turns out that attributes on cdef functions that are used outside the class have to be explicitly declared public in Cython.  Well, at least I figured out what was wrong!

More Cython

I've made substantial progress in getting the code to work with Cython, but I'm still working through all the little errors that crop up with I try to optimize the code for it.

I'm going to write something here about Cython, a language I am not yet completely proficient in, so please take the following with that caveat in mind:

The idea behind Cython is that it's basically a C translator for Python.  It takes source files written in Python but with the extension pyx and, using another Python script that functions a bit like a Makefile, first converts them into C source files (with a .c extension) then converts those into Python compiled modules (.pyd) that a Python interpreter can then import.

If you just rename your Python files to .pyx and write a plain setup.py to convert them, then they should work as-is.  In my case they do, and the result is an about 10-20% speedup, not bad but nothing to write home about.

But one thing you can do with Cython is add keywords to variables, functions and classes that impart type information to them, which supposedly helps tremendously with heavy processing-focused applications, applications such as cellular automation.  You can also define a class so that, instead of using a Python dictionary to hold attributes, you use a C struct, by using the cdef keyword when defining the class.  These things are basically what I've spent the past few days of work on In Profundis doing.

Unfortunately, accessing a cdef class's attributes from outside the class appears to work weirdly.  Sometimes it seems to work, but sometimes it throws up a runtime error that it can't find the attribute.  For it to work consistently, apparently, I have to write accessor functions for those attributes.  And sometimes (not all the time) even those accessor functions aren't found.  (I think it might have to do with the return type of the function, which I might have to declare.  I should look into that.)

Anyway, just wanted to keep you all posted.  In Profundis work has been faster this past week than it's been for a long time.  It's having to share time with a for-pay game project I'm working on (which also has to do with celular automation), but it's continuing fairly well.  Expect more updates soon.

Frame rate timings

From faster to slowest:

Psyco
Mixture of selectively-compiled 32-bit Python bytecode and Cython
32-bit Python bytecode
64-bit Python bytecode

Things left to try:
PyPy (may or may not be compatible with Pygame)
Optimized Cython

Installing a working Cython on Windows with MinGW: a comic tragedy

The following is the contents of a text file I have written to myself and will keep on-hand for the next time I have to install Cython:

---

Setting up Cython:
1. Works with 32-bit Python for some reason.  (I discovered why but forgot.)
2. Install 32-bit Python
3. Install 32-bit MinGW
3. Install setuptools for Win32
4. Write a batch file with the necessary paths, like containing:

*********
SET PATH=C:\MinGW\bin;C:\MinGW\MSYS\1.0\local\bin;C:\MinGW\MSYS\1.0\bin;C:\Python27_32;C:\Python27_32\Scripts
start cmd
*********

5. IN C:\(pythonpath)\Lib\distutils, create distutils.cfg, containing the text:

*********
[build]
compiler = mingw32
*********

6. In cygwinccompiler.py in the same directory, remove all instances of "-mno-cygwin".  This refers to a command line switch that has been removed in more recent versions of Cygwin/MinGW.

7. Run this to set up a command prompt window with the appropriate paths, then run: easy_install cython

8. Write a setup.py for the source modules to compile.  The modules should have the extension .pyx (instead of .py).  It should look something like:

**********
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

ext_modules = [Extension("hello", ["hello.pyx"])]
// ext_modules can contain as many Extension entries as there are
//   modules to compile.  Change the module name and filename to
//   match each module to compile, of course.

// In the following, change name to something appropriate.
setup(
  name = 'Hello world app',
  cmdclass = {'build_ext': build_ext},
  ext_modules = ext_modules
)
**********

9. Run from the pathed command window:
python setup.py build_ext --inplace

The module should now be importable.  It should work, at least.

---

The tragic/comic part of it all is the week it took me to get all this working. I actually did it all for the 64-bit version of Cython and almost got it working, only to be stopped by (yet another) mysterious error, only this one didn't have a solution. There's still one or two steps I ran into doing that that I haven't run into with the 32-bit version yet -- I'll probably run into those once I actually bring Cython to In Profundis' code (currently I've only compiled Hello World with it).

Progress 4/2

Back into the swing of things....

So I've been researching NumPy, a Python module for doing fast things with arrays, to see if it might help speed things up a bit.  Looks promising, although to get the speed advantage I'll have to redesign the code somewhat to use parallel arrays instead of a single 2D array of objects.

I discovered, with Psyco, the code is slight more than twice as fast as with it off.  On, I tend to get framerates in the 40-50 range, off, around 22-25.  But alas, Psyco's homepage now lists the project as officially abandoned, which is kind of a blow honestly.  Its successor PyPy isn't universally useable yet -- in particular, it's incompatible with Pygame.

Another possibly useful modules/packages is Cython, which is a more explict method of compiling Python code, but has the disadvantage that the best optimizations requires peppering the code with variable type declarations, which are not legal Python in themselves, so it's not a drop-in system.