Since part I, we're having a empty Magento2 demo webshop (Luma) successfully running on the new VPS and that means it's time to start working on the real migration from Magento 1 to 2. The magento2 docs page describes the data migration as a straight forward process, let's see if it is true.
I think it's partly true and it started that this part of the migration was really a lot of work of experimenting, doing and re-doing. Things like tables created by plugins are not automatically covered by the tool. On the other hand this gained insights in how the migration tool works and that for example a migration only can be done once. If you want to do it a second time, you need to install a clean database what is easier done in dropping the database and reinstall Magento 2 to start over.
The delta functionality provided by the migraion tool can for work for a short time, but I've experienced that in my case it wouldn't work for days / weeks. This challenge would mean to do the migration over and over again, becuase you simple can't do the migration twice in an active shop situation. If you still would try, you get errors which are unresolveable.
I've have written down al steps taken to do the an one time migration and tried to keep this guide as general as possible. Data migrations are very specific to the webshop. It depends for example on how many plugins you're using or if you did adjust the database by adding columns, etc. If you would follow this guide, you should expect to do ohther modifications (mappings) not described in the guide mostly on the maps.xml file.
I've also written bash scripts to automate all steps. I will post this script snippets on my bitbucket account if you would like this?

Demo site of Magento2

The Migration

According to the Magento DevDocs site, which you can find here, there are 4 sections to migrate:

  • Settings
  • Data
  • Theme
  • Plugins

This guide will focus on the firs two being settings and data.

Get the tooling

Downloading and installing the Magento data migration tool for the version we've got installed on our machine. Browse to your magento directory and check which version you have running. Each version of Mangento2 requires a different version of the migration tool.

$ cd /usr/share/nginx/magento-webshop.nl/httpsdocs/
$ sudo -u magento_user php bin/magento --version
Magento CLI version 2.2.2

The Magento setup is running as the magento_user, member of the www-data group and this gives conflicts when being authenticated as yourself. In this guide we're often going to switch roles. De magento_user is limited and can't do any root activities. The next following steps will add the migration tool to the installed Magento codebase. I'm a fan of the 'composer-way' and that means for you that every step in this guide is done with composer.

$ sudo su - magento_user
$ composer config repositories.magento composer https://repo.magento.com
$ composer require magento/data-migration-tool:2.2.2
$ exit

Open-up the Mangeto1 database for external access

To be able to migrate the Magento1 data, we need to allow a connection from the Magento1 database on the "current" VPS[1]. During production for security reasons only localhost connections are allowed to access the database and UFW is blocking the normal used port 3306. Both are very good things you almost must have in place. Login to the Magento1 VPS and follow the following steps to allow remote access from a singele IP address to the database. First add the migration user with password to Magento1 database in MySQL.

$ ssh your_username@you.re.ip.add.ress
$ mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Server version: 5.7.17 MySQL Community Server (GPL)

mysql> SHOW DATABASES;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mage1_database     |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.01 sec)

mysql> GRANT ALL ON mage1_database.* TO mage_migration@’37.21.243.179’ IDENTIFIED BY ‘password123456’;

Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

mysql> \q
Bye

The next step is to open the firewall to allow port 3306.

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere
80                         ALLOW       Anywhere
443                        ALLOW       Anywhere
22 (v6)                    ALLOW       Anywhere (v6)
80 (v6)                    ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)

$ sudo ufw allow mysql
Rule added
Rule added (v6)

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere
80                         ALLOW       Anywhere
443                        ALLOW       Anywhere
3306                       ALLOW       Anywhere
22 (v6)                    ALLOW       Anywhere (v6)
80 (v6)                    ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)
3306 (v6)                  ALLOW       Anywhere (v6)

Followed by letting MySQL listen not only to localhost, but also other IP addresses. The line bind-address = 127.0.0.1, needs to be commented out.

$ cd /etc/mysql/
$ sudo nano my.cnf

# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address           = 127.0.0.1

Restart the MySQL service to run with the new setting.

$ sudo service mysql restart
 * Stopping MySQL Community Server 5.7.17
...
 * MySQL Community Server 5.7.17 is stopped
 * Re-starting MySQL Community Server 5.7.17
..
 * MySQL Community Server 5.7.17 is started

The above steps are the settings that are required to change. We can close now the open connection with the Magento1 server. The current server is a Ubuntu 14.04 server and doesn't has systemctl to manage services. This is why we've using different methods of managing the services.

Backup current Magento2 database

Before we start to migrate it's common sense to backup the original installed empty database, in case the migration fails and you easy need to restore the original database. The learning about this is that it take less steps and is faster to reinstall Magento2, then restoring the database ;-) I litterally did this once.

$ cd ~
$ mysqldump --opt --user=root --password magento2_database > magento2_database.backup.sql

Configuring the migration

Before you migrate any data, you must create a config.xml configuration file from the provided default sample (config.xml.dist). Please read carefully through the Magento DevDocs about how to configure the migration topic. Magento provides a complete flow diagram that show each step to take. migration_flow Each shop is different and migration tool requires that all documents, fields, etc., are managed. This can be from a simple ingnore, rename, migrate, move, transform and more. Both the source and the destination need to be defined. The mappings are set in de map.xml which also needs to be created. You can find an example in the version of Magento which is the source of your migration. In other words same location as te (config.xml.dist)

To create and edit a configuration file:

$ cd /usr/share/nginx/magento-shop.nl/httpdocs/vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.3.0
$ sudo su - magento_user
$ cp config.xml.dist config.xml
$ cp map.xml.dist map.xml
$ nano config.xml
$ exit

The minimal following settings need to be adjusted in the config.xml:

<source>
    <database host="source-server-ip" name="database_magento1" user="root" password="pass"/>
</source>
<destination>
    <database host="127.0.0.1" name="database_magento2" user="root" password="pass"/>
</destination>
<options>
    <source_prefix>magento1</source_prefix>
    <crypt_key>w3e24aee619dae23887df9fs594f01985</crypt_key>
</options>

After this you can do a run to see what documents & fields need to ne mapped. The migration tool does many intergration checks and stops at any error. To run the migration see the next chapter.

The first time you run it you will see a lot of errors like "Source fields not mapped", "Destination fields not mapped.", "Source documents not mapped" etc.
Referred as "document" here is a database table, a "field" is a column.
You should ignore all the fields that are listed in "Source fields not mapped" errors. For example lets take this error:

[ERROR]: Source fields not mapped. Document: sales_flat_order_address. Fields: fooman.totals

To solve this you will have to add the following to the map.xml

<source>
  <ignore>
    <field>sales_flat_order_address.fooman.totals</field>
  </ignore>

You solved the error "Source fields not mapped. Document: sales_flat_order_address. Fields: fooman.totals"

Hereby I would strongly suggest to create a backup of the two edited files on for example your home folder. On each composer update/upgrade of the migration tool the created files are deleted, which means you will loose all your custom settings.

Start the migrations!

To start the migration you need to follow the next steps in exact order:

1. Disable all crontabs (crontab -e) and comment any Magento releated lines out

$ sudo su - magento_user
$ cd /usr/share/nginx/magento-shop.nl/httpdocs/
$ php bin/magento cron:remove
$ exit

2. Ensure all files & folders are owned by the magento_user and have the correct rights

$ sudo chown magento_user:www-data * -R
$ sudo su - magento_user
$ find var vendor pub/static pub/media app/etc -type f -exec chmod g+w {} \;
$ find var vendor pub/static pub/media app/etc -type d -exec chmod g+ws {} \;
$ chmod u+x bin/magento

I'm not closing the mangento_user session here, because you need it soon a couple times more.

3. Put the Magento1 website into maintenance mode

All traffic during migration is bad for the migration because of the mitations happing on the data.
In your favorite terminal open a second windows/tab an login to your VPS. In order to put site to maintenance mode you should create empty maintenance.flag file and upload it to the root folder

$ ssh your_username@you.re.ip.add.ress
$ cd /usr/share/nginx/your-mangento1-shop.nl/httpdocs/
$ touch maintenance.flag
$ exit

Check by browsing to the Magento1 webshop if the maintenance mode really is enabled!

maintenance-mode

4. Put the Magento2 webshop in maintenance mode.

$ php bin/magento maintenance:enable
Enabled maintenance mode
$ exit

You can double check by browsing to the webshop url, but the Magento2 CLI[2] gives already feedback on the commandline.

5. On the Magento1 server clear the all caches & log files.

I've posted a snippet on Bitbucket. Put the cleanup.php file in the webroot of the current Magento 1 website. Before running the cleanup, everybody must logout from the admin section and the Mangento 1 shop must be in maintenance mode.
Then run first https://www.magento-one.nl/cleanup.php?clean=log & then secondly https://www.magento-one.nl/cleanup.php?clean=var. From now on do not login into the admin section and do not use the Magento1 webshop.

6. Move the media folder from Magento 1 to Magento 2

Development is done here on a MacBook. To move and cleanup the /media folder from the Mangento1 shop I'm using a simple FTP program called Transmit. Within Transmit I start a connection over SFTP[3] to the VPS and copy the folder to my local development machine.
After this is done. I've cleaned up the folder (e.g. removed files from plugins not needed anymore, caches, etc) and copied it back to the home folder of the Magento2 VPS. Due to good security reasons I can only create files on my home folder. After this is done just move the files to the new Media folder:

$ sudo cp -R ~/media/* /pub/media/
$ sudo chown magento_user:www-data * -R

7. Migrate the settings, reindex, clean & flush the caches

$ sudo su - magento_user
$ php bin/magento migrate:settings --reset /usr/share/nginx/magento-shop.nl/httpdocs/vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.3.0/config.xml

$ php bin/magento indexer:reindex
$ php bin/magento cache:clean
$ php bin/magento cache:flush

Reindexing and cleaning the cache are a precautionary measure.

8. Migrate the data, reindex, clean & flush the caches

$ php bin/magento migrate:data --reset /usr/share/nginx/magento-shop.nl/httpdocs/vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.3.0/config.xml

$ php bin/magento indexer:reindex
$ php bin/magento cache:clean
$ php bin/magento cache:flush

Reindexing and cleaning the cache are now a required step!

9. Remove all generated files.

$ rm -rf /usr/share/nginx/magento-shop.nl/httpdocs/var/cache/*
$ rm -rf /usr/share/nginx/magento-shop.nl/httpdocs/var/page_cache/*
$ rm -rf /usr/share/nginx/magento-shop.nl/httpdocs/generated/* 
$ rm -rf /usr/share/nginx/magento-shop.nl/httpdocs/var/di/* 
$ rm -rf /usr/share/nginx/magento-shop.nl/httpdocs/var/cache/* 
$ rm -rf /usr/share/nginx/magento-shop.nl/httpdocs/var/view_preprossed/* 
$ rm -rf /usr/share/nginx/magento-shop.nl/httpdocs/pub/static/* 
exit

After you've finished the settings, we now can setup (or just (re-)enable) the automatic indexers with the help of cronjobs.

Cronjobs

The cronjob should run every minute under the magento_user user. This advised and described in the Mangento documentation. Having Magento version 2.2 running, this is fairly simple.

$ sudo su - magento_user
$ cd /usr/share/nginx/magento-shop.nl/httpdocs/
$ php bin/magento cron:install [--force]

Use --force to rewrite an existing Magento crontab.

To view the crontab, enter the following command as the Magento file system owner:

crontab -l

The last step of the data migration

When all data is migrated and all after-migration steps are taken it's time to take the Magento2 webshop out of maintenance mode, start testing.

$ php bin/magento maintenance:disable
Enabled maintenance mode
$ exit

After having the webshop running we can remove the migration tool. This is not needed anymore. To do this:

$ composer remove magento/data-migration-tool:2.2.2
$ composer update
$ php bin/magento setup:upgrade
$ php bin/magento setup:di:compile
$ php bin/magento setup:static-content:deploy -fl

And that's it!

In the next blogpost we'll describe the last two missing parts. This will cover the theme and the plugins migration. Thanks for reading and I hope it will help you. I've you got any feedback let me know in the comments!


  1. VPS = Virtual Private Server ↩︎

  2. CLI = Command Line Interface ↩︎

  3. SFTP = SSH File Transfer Protocol ↩︎