Transmogrify 100k AT objects to Dexterity types
Since a while we are working on our new support system. Our support system supports our workflows.
It holds all our customer information, data, project information and support tickets are registered by us and our customers in our support system. Using the tickets, we can communicate with our customers and keep track of the status of the issues.
Support was built around five years ago, written in Plone3, based on Poi. It is time for an upgrade, since our working methods have evolved, and, we like to update to Plone 4. We don't want to lose our existing tickets, so we're migrating all the data from the old support system to the new support system.
We are exporting around 100.000 objects to JSON files from the old support system. We are using collective.blueprint.jsonmigrator to do this. Each JSON file represents an ATContentType Plone-object, with all the properties we need (now or possibly in the future) to keep all the existing data. This process takes about 30 minutes to run. After this, we have 100.000 files including all data and blob files.
In the new support system we are importing the objects from the exported JSON files. For this we're using the collective.transmogrifier and transmogrify.dexterity packages. Several passes are performed on each JSON file to convert it to, for example Dexterity types, and change properties where needed. We read the Title of the object from the JSON file and store it in the newly created object.
Another step is changing the IDs of the tickets. In the old system, the IDs are simple integers, which are increased by one, each time a ticket is created. Each ticket belongs to a Tracker and there are several Trackers, at least one for each customer. In the new support system we are using a new ID-creation scheme where each ticket has a globally unique ID.
The new support system is based on bootstrap responsive layout and has the plone.app.toolbar for editing. All contenttypes are dexterity types, so it includes plone.app.contenttypes for folders and documents.
The whole import-process takes around 2 hours. So we are exporting and importing around 100.000 objects in less than 3 hours.
