Writing your first Drupal migration

Submitted by dinarcon on Fri, 08/02/2019 - 09:00

Last updated on March 2, 2020.

In the previous entry, we learned that the Migrate API is an implementation of an ETL framework. We also talked about the steps involved in writing and running migrations. Now, let’s write our first Drupal migration. We are going to start with a very basic example: creating nodes out of hardcoded data. For this, we assume a Drupal installation using the `standard` installation profile, which comes with the `Basic Page` content type. As we progress through the series, the migrations will become more complete and more complex. Ideally, only one concept will be introduced at a time. When that is not possible, we will explain how different parts work together. The focus of today's lesson is learning the structure of a migration definition file and how to run it.

List of migrate plugins

Writing the migration definition file

The migration definition file needs to live in a module. So, let’s create a custom one named `ud_migrations_first` and set Drupal core’s `migrate` module as dependencies in the *.info.yml file.

type: module
name: UD First Migration
description: 'Example of basic Drupal migration. Learn more at <a href="https://understanddrupal.com/migrations" title="Drupal Migrations">https://understanddrupal.com/migrations</a>.'
package: Understand Drupal
core: 8.x
  - drupal:migrate

Now, let’s create a folder called `migrations` and inside it, a file called `udm_first.yml`. Note that the extension is `yml` not `yaml`. The content of the file will be:

id: udm_first
label: 'UD First migration'
  plugin: embedded_data
      unique_id: 1
      creative_title: 'The versatility of Drupal fields'
      engaging_content: 'Fields are Drupal''s atomic data storage mechanism...'
      unique_id: 2
      creative_title: 'What is a view in Drupal? How do they work?'
      engaging_content: 'In Drupal, a view is a listing of information. It can a list of nodes, users, comments, taxonomy terms, files, etc...'
      type: integer
  title: creative_title
  body: engaging_content
  plugin: 'entity:node'
  default_bundle: page

The final folder structure will look like:

|-- core
|-- index.php
|-- modules
|   `-- custom
|       `-- ud_migrations
|           `-- ud_migrations_first
|               |-- migrations
|               |   `-- udm_first.yml
|               `-- ud_migrations_first.info.yml

YAML is a key-value format with optional nesting of elements. It is very sensitive to white spaces and indentation. For example, it requires at least one space character after the colon symbol (:) that separates the key from the value. Also, note that each level in the hierarchy is indented by two spaces exactly. A common source of errors when writing migrations is improper spacing or indentation of the YAML files.

A quick glimpse at the file reveals the three major parts: source, process, and destination. Other keys provide extra information about the migration. There are more keys than the ones shown above. For example, it is possible to define dependencies among migrations. Another option is to tag migrations so they can be executed together. We are going to learn more about these options in future entries.

Let’s review each key-value pair in the file. For the `id` key, it is customary to set its value to match the filename containing the migration definition, but without the `.yml` extension. This key serves as an internal identifier that Drupal and the Migrate API use to execute and keep track of the migration. The `id` value should be alphanumeric characters, optionally using underscores (_) to separate words. As for the `label` key, it is a human readable string used to name the migration in various interfaces.

In this example, we are using the `embedded_data` source plugin. It allows you to define the data to migrate right inside the definition file. To configure it, you define a `data_rows` key whose value is an array of all the elements you want to migrate. Each element might contain an arbitrary number of key-value pairs representing “columns” of data to be imported.

A common use case for the `embedded_data` plugin is testing of the Migrate API itself. Another valid one is to create default content when the data is known in advance. I often present Drupal site building workshops. To save time, I use this plugin to create nodes which are later used when explaining how to create Views. Check this repository for an example of this. Note that it uses a different directory structure to define the migrations. That will be explained in future blog posts.

For the destination, we are using the `entity:node` plugin which allows you to create nodes of any content type. The `default_bundle` key indicates that all nodes to be created will be of type “Basic page”, by default. It is important to note that the value of the `default_bundle` key is the machine name of the content type. You can find it at `/admin/structure/types/manage/page`. In general, the Migrate API uses machine names for the values. As we explore the system, we will point out when they are used and where to find the right ones.

In the process section, you map columns from the source to node properties and fields. The keys are entity property names or the fields' machine names. In this case, we are setting values for the `title` of the node and its `body` field. You can find the field machine names in the content type configuration page: `/admin/structure/types/manage/page/fields`. During the migration, values can be copied directly from the source or transformed via process plugins. This example makes a verbatim copy of the values from the source to the destination. The column names in the source are not required to match the destination property or field name. In this example, they are purposely different to make them easier to identify.

The repository, which will be used for many examples throughout the series, can be downloaded at https://github.com/dinarcon/ud_migrations Place it into the `./modules/custom` directory of the Drupal installation.  The example above is part of the “UD First Migration” submodule so make sure to enable it.

Running the migration

Let’s use Drush to run the migrations with the commands provided by Migrate Run. Open a terminal, switch directories to Drupal’s webroot, and execute the following commands.

    • $ drush pm:enable -y migrate migrate_run ud_migrations_first
    • $ drush migrate:status
    • $ drush migrate:import udm_first

Note: It is assumed that the Migrate Run module has been downloaded via composer or otherwise.

Important: All code snippets showing Drush commands assume version 10 unless otherwise noted. If you are using Drush 8 or lower, the commands’ names and aliases are different. Usually, a hyphen (-) was used as delimiter in command names. For example, `pm-enable` in Drush 8 instead of `pm:enable` in Drush 10. Execute `drush list --filter=migrate` to verify the proper commands for your version of Drush.

The first command enables the core migrate module, the runner, and the custom module holding the migration definition file. The second command shows a list of all migrations available in the system. For now, only one should be listed with the migration ID `udm_first`. The third command executes the migration. If all goes well, you can visit the content overview page at /admin/content and see two basic pages created. Congratulations, you have successfully run your first Drupal migration!!!

Or maybe not? Drupal migrations can fail in many ways. Sometimes the error messages are not very descriptive. In upcoming blog posts, we will talk about recommended workflows and strategies for debugging migrations. For now, let’s mention a couple of things that could go wrong with this example. If after running the `drush migrate:status` command, you do not see the `udm_first` migration, make sure that the `ud_migrations_first` module is enabled. If it is enabled, and you do not see it, rebuild the cache by running `drush cache:rebuild`.

If you see the migration, but you get a YAML parse error when running the `migrate:import` command, check your indentation. Copying and pasting from GitHub to your IDE/editor might change the spacing. An extraneous space can break the whole migration so pay close attention. If the command reports that it created the nodes, but you get a fatal error when trying to view one, it is because the content type was not set properly. Remember that the machine name of the “Basic page” content type is `page`, not `basic_page`. This error cannot be fixed from the administration interface. What you have to do is rollback the migration issuing the following command: `drush migrate:rollback udm_first`, then fix the `default_bundle` value, rebuild the cache, and import again.

Note: Migrate Tools could be used for running the migration. This module depends on Migrate Plus. For now, let’s keep module dependencies to a minimum to focus on core Migrate functionality. Also, skipping them demonstrates that these modules, although quite useful, are not hard requirements to work on migration projects. If you decide to use Migrate Tools, make sure to uninstall Migrate Run. Both provide the same Drush commands and conflict with each other if the two are enabled.

What did you learn in today’s blog post? Did you know that Migrate Plus and Migrate Tools are not hard requirements for Drupal migrations projects? Did you know you can place your YAML files in a `migrations` directory? What advice would you give to someone writing their first migration? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with your friends and colleagues.


Kieran Mathieson (not verified)

Sun, 08/04/2019 - 09:08

Thanks for writing this. I'm looking forward to learning more about migrations.

Kurppa-Jooseppi (not verified)

Sun, 10/06/2019 - 15:27

Thank you for very useful and clear content.

Just a small thing, In the section:
"execute the following commands.
• $ drush pm:enable -y migrate migrate_run ud_migrations_first"

It wont work, because user has to first download the "migrate_run" -module. At least in my system that module was not available.

Hi Kurppa-Jooseppi. Yes, the Migrate Run module has to be downloaded prior executing the commands. That can be done via composer or another method. A note has been added to clarify this. Thanks for your comment!

Hi Cherie, the command names depend on the version of Drush being used. The article assumes the latest stable version Drupal 10. The commands you shared corresponds to Drush 8. A note has been added to clarify this point. Thanks for your comment!

Doomd (not verified)

Thu, 06/25/2020 - 15:03

I think going through your 31 days articles before attempting a major migration will actually save ALOT of time. I've already wasted many days fumbling my way through on a 50,000+ 30-column migration...and there are too many errors to list here...but I decided to bite the bullet and read all these articles first so I'm not shooting in the dark with blanks. Thanks for the bullets.


Thu, 06/25/2020 - 16:25

In reply to by Doomd (not verified)

Thanks for your comment Doomd. We hope the series proves to be useful. It would be great to read your comments on the different articles as you progress through the series. Also make sure to check the migrations tag for more articles on the same topic https://understanddrupal.com/tag/migrations

Amir Simantov (not verified)

Sun, 08/30/2020 - 08:17

This is a great tutorial, thanks.

I was following the process of the import with a debugger and stopped in the middle. So, it seems that I left something half-baked and couldn't run rollback nor running import again.
The error that I got was: Migration udm_first is busy with another operation: Importing
The solution was to run this command: drush migrate-reset-status --all

Going on to the next chapters...

Stephen Cross (not verified)

Thu, 05/06/2021 - 08:39

Thanks for putting this together. I'm going through the days with Drupal 9. A few changes that others my interested in related to day 2:

1. drush 10.4.x includes the commands provided by migrate_run module, I don't believe migrate_run is needed.

2. Update ud_migrations_first.info.yml by replacing "core: 8.x" with "core_version_requirement: 9.x"


Wed, 05/12/2021 - 16:09

In reply to by Stephen Cross (not verified)

Thanks for your comment Stephen. We are planning to update the series to include the latest developments. Indeed, with Drush 10.4 there is no need for Migrate Run. As for the examples, many of them are already compatible with Drupal 8 and 9. A new are pending to be updated though.

David (not verified)

Mon, 05/10/2021 - 15:00

Thanks for this amazing series!

Just a heads up that the functionality provided by the Migrate Run module has been moved to Drush core (since v10.4.0).

Add new comment

Plain text

  • No HTML tags allowed.
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.