So these past few weeks I've been working hard on a system to handle importing items from numerous third party providers without using a huge amount of resource.
Initially this seemed a trivial task and I set out with much hope of completing this within the week. Turns out it's not as easy as I thought.
However, 2 months later I have now completed my task.
I've managed to take a system that historically took around 2 hours and consumed 2GB of RAM on the host server and recreated this using modern technologies and my knowledge, into a system that now consumes 30MB of RAM maximum (fluctuates between 15 and 30MB) and takes under 20 minutes to download the data, prices it and insert it into the database.
All this is running on my local machine, on a slower than production connection. My local dev machine runs PHP5.3.6 and MySQL 5.1.
I have some plans on refactoring parts of the application and perhaps publishing some guides/information about the techniques and libraries I used to create this application, but for now below is a list of things i used to create this: