Data Import for Firestore

Overview

The document import functionality in Fuego allows users to seamlessly import structured data into Firestore collections. It supports various file formats, including CSV, JSON, JSONL and YAML, providing options for document creation, updating, and merging.

Import Process

1. Select File: Choose the file to import. The format is detected from the file extension. Supported formats:

   • CSV (with automatic type conversion for numeric, boolean, and null values, though inconsistencies may arise)
   • JSONL (recommended for best compatibility and consistent formatting)
   • JSON, either an array of documents or an object keyed by document path
   • YAML (.yaml, .yml), the same shapes the YAML export produces (see below)

2. Destination: Import to project and database starts on the project and database you are viewing, and can be changed to send the file anywhere else you have configured. Picking a different one shows a warning — the collection and attribute lists then describe that destination, and the preview and the job both run against it.

3. Import Options:

   • Create new documents if not present: If enabled, new documents will be generated whenever an ID is not found in the existing collection.
   • Use __path__ field: If present, this field dictates document paths within Firestore’s hierarchy, allowing for structured organization.
   • Alternative Document ID Field: If specified, this field serves as a fallback identifier when __name__ is unavailable. This ensures flexibility in managing unique document IDs.

4. Update Modes:

   • Overwrite: Completely replaces existing documents, discarding any previous data.
   • Merge: Integrates new data into existing documents while retaining unchanged fields. This method is useful for incremental updates.
   • Update: Modifies only explicitly designated fields without affecting the rest of the document. This is ideal for targeted field-level modifications.

Additional Settings

• Attribute Handling:
  • Retain all attributes included in the import file, ensuring completeness in data import.
  • Exclude specific attributes from the import process when creating or updating documents to avoid unintended changes.
• Special Value Conversions:
  • Convert specific string values, such as timestamps, GeoPoints, and Firestore references, to their appropriate data types.
  • Utilize special placeholders such as __DELETE__ to remove attributes dynamically or __SERVER_TIMESTAMP__ for automatic timestamping during import.
• CSV-Specific Settings:
  • Define the delimiter used in the CSV file. The default delimiter is a comma (,), but users can modify this based on their file structure.
  • Handle inconsistent data formatting by ensuring all fields adhere to Firestore’s expected data types.
• JSON and YAML Settings:
  • When the file is a mapping keyed by document path, enable Use the key as the document path so that every document lands where its key says. Otherwise the documents go into the selected collection.

Transform Script

Enable Transform each document with a custom script to rewrite every row of the file with JavaScript before it is imported. The script runs inside the app, once per row, and receives:

• docId — the ID of the document about to be written
• collection — the target collection path
• data — the raw row from the file, including Fuego entity values such as {"__time__": …}
• the common globals — db, database(), fetch, util, the object helpers get / pick / omit, … — described in the Custom Scripts reference

Return the data to import — using JavaScript natives such as new Date() or the entity maps, whichever is more convenient — or undefined to skip the row:

// Stamp each imported row
return { ...data, importedAt: new Date(), source: 'legacy-csv' };

The import preview runs the script too, so you can check the transformed rows before confirming — database writes are disabled while previewing.

The If the script fails on a document setting decides whether a failing row is skipped (the default — failures are listed in the job details) or stops the import. Scripts can be saved to the library and reloaded with Load script… — see Custom Scripts for the full contract and limits.

YAML Files

A YAML import accepts the file the YAML export writes, a mapping where each key is the document path:

---
users/alice:
    name: Alice
    age: 30

users/bob:
    name: Bob
    tags:
        - admin
        - beta

Two hand-written shapes are accepted as well: a sequence of documents, and a stream of documents separated by ---. In those shapes the document path comes from the __path__ field, from the document ID field, or from the selected collection.

- __path__: users/alice
  name: Alice
- __path__: users/bob
  name: Bob

Import Warnings

• File Format Considerations: CSV imports may lead to inconsistencies due to type conversion limitations. JSONL is recommended for optimal data integrity.
• Type Conversion Rules: Automatic conversions apply to numeric, boolean, and null values in CSV files. Users should verify that their data structure aligns with Firestore’s expectations.
• Handling Special Values: Reserved placeholders such as __DELETE__ and __SERVER_TIMESTAMP__ will always be interpreted according to Firestore’s specifications, ensuring appropriate data transformations.

Finalizing the Import

Once all settings have been configured to the user’s preference, clicking Confirm will initiate the import process. Users can monitor progress and review imported data within the Firestore database. Additionally, to streamline future imports, preferences can be saved for reuse, reducing configuration time and ensuring consistency across multiple data import sessions.