This is a two parts blog series: Part 1 is focused on a simple command to export/import data, while Part 2 handles with more advanced use cases.
Every time I begin a new project for a customer and that we start talking about using Scratch Orgs, one of the question that keeps coming up is “How do I load data ?”.
What’s already available
The Salesforce CLI comes with a nice set of commands to export data,
sfdx force:data:tree:export and its counterpart to import
sfdx force:data:tree:import. These commands are pretty simple, you just have to pass an SOQL query for the data you want to export, but comes with 2 main issues:
- as you need an SOQL query to use as an input, this means that the data you can export is tied to the SOQL limitations
- some data are org-specific because of Ids, like User Ids or Record Type Ids. This doesn’t work well with Scratch Orgs where either users don’t exist or Ids will change for every new Scratch Org
At Texeï, our whole development workflow is based on Scratch Orgs, so we quickly ran into this second issue. For each new Scratch Org, Record Types were recreated, which means different Ids were generated. At first we were just importing data without Record Type, and manually updating records to select the correct record type.
This quickly became painful. Enters custom CLI plugins.
What we’ve built
We decided to build our own import and export tool to make our own life easier. Note that these commands are not supposed to solve all use cases and were build with some principles in mind:
- be simple to use
- manage record types and user ids
- be focused on Scratch Org, or at least development environment
- be easy to export the same data again and “refresh” the dataset
There are already out there lots of tools to handle very specific and complicated use cases with data. We wanted an easy one to manage all the data needed in a development environment. The flow is the following:
- Create a Scratch Org
- Create the data you need, once
- Export all the data
So how do I export the data you’ll ask ? Just create a folder, let’s call it
data and run a command like this:
sfdx texei:data:export --objects Account,Contact,MyCustomObject__c --outputdir ./data --targetusername MyOrg
Then, you’ll just have to run an import command to insert the exact same data in your next Scratch Org:
sfdx texei:data:import --inputdir ./data --targetusername MyOtherScratchOrg
As you can see, ease of use is the main focus on these commands. Exporting is just a matter of listing the sObjects you want, the only requirement is to list them in the correct order, which is if a Contact references an Account, accounts must be exported first. The plugin will handle Record Types for you, and use the Scratch Org user for lookups referencing users.
No filters on the records, no fields selection. And this is on purpose. Do the job once and you’re good to go.
Note: With data privacy you’ll likely have to work on your test data or create some dummy ones anyway, so filtering records was not something we wanted to implement. Also, if you don’t have a way to add data to a development environment yet, you’ve likely created test data in your sandboxes manually, so it’s just the matter of doing it one last time.
How do I install it ?
If you haven’t used Texeï’s plugin yet, installing it is damn simple:
- install Salesforce CLI if you haven’t already
- install Texeï’s plugin:
sfdx plugins:install texei-sfdx-plugin
Note that the plugin is open source, and available on GitHub. Pull Requests welcomed 😊
Life isn’t always that simple
This is super easy to use when starting a new project or doing a pre-sales (because yes, we’re doing our pre-sales demos on Scratch Orgs 🤩), but all data models are not that easy to populate.
We’ll see how to handle more complex use cases in the second part of this blog post serie (coming soon).